Method and system for providing image-based interoperability with an application

ABSTRACT

A system and method for enabling graphic-based interoperability between computer executed applications. A computer system operating as a client may display a graphical user interface (GUI) including control graphic items such as buttons, text boxes, etc. A process may examine the graphical image of the GUI to determine if there has been a change over time in the GUI as displayed which updates a control graphic item. If there has been a change over time in the GUI which updates a control graphic item, an action may be taken, for example updating properties of an object construct corresponding to the control graphic item, raising an event corresponding to an object construct corresponding to the control graphic item, or communicating an event to a process.

RELATED APPLICATION DATA

The present application is a continuation-in-part of application Ser.No. 16/701,299 filed Dec. 3, 2019, entitled “IMAGE BASED METHOD ANDSYSTEM FOR BUILDING OBJECT MODEL AND APPLICATION STATES COMPARISON ANDGRAPHIC-BASED INTEROPERABILITY WITH AN APPLICATION”, which in turn is acontinuation of prior US application Ser. No. 15/921,705, filed Mar. 15,2018, entitled “IMAGE BASED METHOD AND SYSTEM FOR BUILDING OBJECT MODELAND APPLICATION STATES COMPARISON AND GRAPHIC-BASED INTEROPERABILITYWITH AN APPLICATION” which in turn is a continuation-in-part of priorU.S. application Ser. No. 15/416,484 entitled SYSTEM AND METHOD FORENABLING GRAPHIC-BASED INTEROPERABILITY WITH A RUN-TIME APPLICATION,filed on Jan. 26, 2017, each of which being incorporated herein in itsentirety.

FIELD OF THE INVENTION

The present invention is in the field of interoperability. Inparticular, the present invention is directed to systems and methods forenabling graphic-based or image-based interoperability between computersystems.

BACKGROUND OF THE INVENTION

Interoperability may be the ability of a computer system or applicationto work with other computer systems or applications, typically withoutspecial effort on the part of the users. Although many applicationsexist which can expose to another application or system an externalgraphical user interface (GUI) for interoperability via an applicationprogramming interface (API) or a software development kit (SDK) whichallow interaction with the GUI by low-level programming techniques,there exist applications which cannot be connected or have theirinternal events easily known by any existing technique.

An example of an application which often cannot be connected by standardAPI and/or SDK techniques is an application executing or running in aremote environment (e.g., in a server/client architecture). In thiscase, the user (e.g. using the client system) may be able to see animage of an application on the screen of the client device and performvarious input operations using a keyboard and/or mouse or touchscreen,but existing application integration techniques cannot recognize userinterface (UI) elements or connect to any API exposed by theapplication, even when such APIs exist and are available or ready touse. As such, a third party application attempting to work with oraccess an application may not be able to. Examples of remoteenvironments in which this problem is prevalent include the MicrosoftRemote Desktop system, the Citrix XenApp system, the PCAnywhere system,and the Oracle VM system.

Interoperability problems can be attributed to one or more of forexample lack of a reliable connector (e.g. API or SDK) for aninteraction with such applications; and lack of an object model exposedby GUI elements or objects (for example, buttons, list boxes, links,tables, etc.).

One prior attempt at solving the interoperability issue includestechnology focused on Optical Mark Recognition (OMR), which is used withOptical Character Recognition (OCR) engines to format text whilegenerating a specific text document. OMR also provides the ability torecognize a text document with a template designed by developers.However, OMR and similar technologies do not provide an ability tointeract with an application image, among other deficiencies.

SUMMARY OF THE INVENTION

A system and method for enabling graphic-based interoperability betweencomputer executed applications. A computer system operating as a clientmay display a graphical user interface (GUI) including control graphicitems such as buttons, text boxes, etc. A process may examine thegraphical image of the GUI to determine if there has been a change overtime in the GUI as displayed which updates a control graphic item. Ifthere has been a change over time in the GUI which updates a controlgraphic item, an action may be taken, for example updating properties ofan object construct corresponding to the control graphic item, raisingan event corresponding to an object construct corresponding to thecontrol graphic item, or communicating an event to a process.

Some embodiments of the invention improve the underlying functionalityof the computer systems on which embodiments of the invention areexecuted by for example allowing different executing programs tocommunicate, integrate, operate together, or work together, or to moreefficiently do so. Embodiments of the invention may improve thetechnology of computer application interoperability and communications.For example, embodiments may enable the development and use of run-timesolutions for integration of an application with third partyapplications, such as applications running in remote environments havingonly visual representation of the UI on a client (e.g., a customer)desktop. Application communications independent of platform, operatingsystem or standards, and based only on a visual representation, may beachieved. Some embodiments of the invention enhance functionality ofapplications instantiated on client devices in a server/clientarchitecture, which may make it possible to avoid unnecessaryinstallation, development, or configuration of software on remoteservers and client devices, thus conserving memory and maximizingprocessing power on such servers and devices. Such enhancements mayreduce total cost and development time for applications to be run insuch environments.

Remote servers provided to customers by external information technology(IT) companies may include security policies and/or physical limitations(memory usage, speed requirements, etc.) which do not allow forinstallation of certain applications and/or software components on theseservers. Some embodiments of the invention solve this problem byproviding the functionality of such applications and/or softwarecomponents without requiring actual installation on such servers, and/orrequiring minimal installation on client systems, which may be moreaccessible to a third party than the server controlling the clientapplication. Furthermore, many applications contain components, some ofthem not standard, that are not accessible via existing integrationtechnologies. Some embodiments of the invention enable access to suchcomponents, which may lower time and effort of research and developmentto develop connectors to support these applications by providing anout-of-the-box solution for it. Embodiments may provide a simple way toadapt or connect two pre-existing software packages. Furthermore, someembodiments of the invention may function as a universal real-timeconnector which may be used with any application type. Such a universalreal-time connector may be independent of the application platform(e.g., the desktop, server, etc.) and even independent of any operationsystem.

According to embodiments of the invention a system and method forenabling graphic-based interoperability with an application executed ona remote computer remote to a local computer including a localprocessor, and displayed on a local display, may include, using thelocal processor: during a design-time mode: receiving a design-timeapplication image and a control object located within the design-timeapplication image; identifying an anchor for the control object; andstoring the design-time control object and the associated anchor; andduring a run-time mode: capturing a run-time application imagedisplaying on the local display; identifying the anchor within therun-time application image; and identifying the control object withinthe run-time application image based on the anchor.

These and other aspects, features and advantages will be understood withreference to the following description of certain embodiments of theinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter regarded as the invention is particularly pointed outand distinctly claimed in the concluding portion of the specification.The invention, however, both as to organization and method of operation,together with objects, features and advantages thereof, may best beunderstood by reference to the following detailed description when readwith the accompanied drawings. Embodiments of the invention areillustrated by way of example and not limitation in the figures of theaccompanying drawings, in which like reference numerals indicatecorresponding, analogous or similar elements, and in which:

FIG. 1 is a high-level diagram illustrating an example configuration ofa system for enabling graphic-based interoperability with a run-timeapplication according to at least one embodiment of the invention;

FIG. 2 is a flow diagram of a first part of a method for enablinggraphic-based interoperability with a run-time application according toat least one embodiment of the invention;

FIG. 3 is an example captured application image, according to at leastone embodiment of the invention;

FIG. 4 is an example captured application image shown, with edges andcontours, converted to grayscale, according to at least one embodimentof the invention;

FIG. 5 is an example set of generated bounding shape objects, accordingto at least one embodiment of the invention;

FIG. 6 is an example shapes tree, according to at least one embodimentof the invention;

FIG. 7 is an example scene object model, according to at least oneembodiment of the invention;

FIG. 8 is a scene design-time data table, according to at least oneembodiment of the invention;

FIG. 9 is a flow diagram of a second part of a method for enablinggraphic-based interoperability with a run-time application according toat least one embodiment of the invention;

FIG. 10 is a control methods, properties and events table, according toat least one embodiment of the invention;

FIG. 11 is a flow diagram for a “designer” stage, for identifying and/ordefining GUI objects in an application image during a design-timelearning stage, according to embodiments of the invention;

FIG. 12 is a flowchart depicting the operation of an RT clientapplication operated side by side with client/server or monitoredapplication on a user terminal, according to one embodiment;

FIG. 13 is a flowchart of a method for enabling graphic-basedinteroperability with an application executed on a remote computer,remote to a local computer including a local processor, and displayed ona local display, according to embodiments of the invention;

FIG. 14 depicts a part of a design-time application image with an OKbutton control, cancel button control, and an anchor identified for thecancel button control, according to embodiments of the invention;

FIG. 15 depicts a part of a design-time application image with anexample of a link control, and anchor identified automatically for thelink control, according to embodiments of the invention;

FIG. 16 depicts a part of a design-time application image with anexample of a check box or radio button control and an anchorautomatically identified for the radio button control, according toembodiments of the invention;

FIG. 17 depicts a part of a design-time application image with anexample of a scrollbar control and an anchor automatically identifiedfor the scrollbar control, according to embodiments of the invention;

FIG. 18 depicts a part of a design-time application image with anexample of scrollbars and scrollbars rectangles, according toembodiments of the invention;

FIG. 19 depicts a part of a design-time application image with anexample of a table control and an anchor automatically identified forthe table control, according to embodiments of the invention;

FIG. 20 depicts a part of a design-time application image with anexample of detected bounding rectangles of words and icons, according toembodiments of the invention;

FIG. 21 is a flowchart of a variant of a method for enablinggraphic-based interoperability with an application executed on a remotecomputer remote to a computer including a local processor, and displayedon a local display, according to embodiments of the invention;

FIG. 22 depicts a part of a design-time application image that isdivided into segments, according to embodiments of the invention; and

FIG. 23 is a flowchart of a variant of a method for enablinggraphic-based interoperability with an application executed on a remotecomputer remote to a local computer including a local processor, anddisplayed on a local display, according to embodiments of the invention.

It will be appreciated that, for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn accuratelyor to scale. For example, the dimensions of some of the elements may beexaggerated relative to other elements for clarity, or several physicalcomponents may be included in one functional block or element. Further,where considered appropriate, reference numerals may be repeated amongthe figures to indicate corresponding or analogous elements.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, various aspects of the present inventionwill be described. For purposes of explanation, specific configurationsand details are set forth in order to provide a thorough understandingof the present invention. However, it will also be apparent to oneskilled in the art that the present invention may be practiced withoutthe specific details presented herein. Furthermore, well known featuresmay be omitted or simplified in order not to obscure the presentinvention.

Although embodiments of the invention are not limited in this regard,discussions utilizing terms such as, for example, “processing,”“computing,” “calculating,” “determining,” “establishing”, “analyzing”,“checking”, or the like, may refer to operation(s) and/or process(es) ofa computer, a computing platform, a computing system, or otherelectronic computing device, that manipulates and/or transforms datarepresented as physical (e.g., electronic) quantities within thecomputer's registers and/or memories into other data similarlyrepresented as physical quantities within the computer's registersand/or memories or other information non-transitory processor-readablestorage medium that may store instructions, which when executed by theprocessor, cause the processor to perform operations and/or processes.Although embodiments of the invention are not limited in this regard,the terms “plurality” and “a plurality” as used herein may include, forexample, “multiple” or “two or more”. The terms “plurality” or “aplurality” may be used throughout the specification to describe two ormore components, devices, elements, units, parameters, or the like. Theterm set when used herein may include one or more items. Unlessexplicitly stated, the method embodiments described herein are notconstrained to a particular order or sequence. Additionally, some of thedescribed method embodiments or elements thereof may occur or beperformed simultaneously, at the same point in time, or concurrently.

Some embodiments of the invention enable graphic-based interaction withGUI elements or controls of an application by for example using shapeanalysis or other graphical analysis of a graphical image of the GUIoutput for an application (which may be termed “application image”,which may vary over time as the application state changes), which can beconsidered the display a human user views as produced by theapplication, to identify graphical objects or controls with whichinteraction can be enabled. Some embodiments of the invention provide aGUI object model and/or algorithms for interaction with GUI elements ofan application recognized as components in application image. Someembodiments of the invention recognize well known graphical objects andgraphical object types such as, for example, application windows, GUIcomponents, controls etc., on, e.g., a computer screen image. For thispurpose, some embodiments of the invention apply a contour and/or shapesanalysis of an application image, e.g., from a captured screen image.

Graphical, visual or GUI objects when discussed herein can typicallyinclude visual representations of on-screen graphics or controls, andmay differ from objects in the programming sense, although programmingobjects can represent the visual graphics objects. Objects in theprogramming or software sense may be, in the class-based object-orientedprogramming model, a particular instance of a class and may be calledfor example instantiated objects, or runtime objects. Graphical or GUIobjects may include for example buttons, icons, menus, cursors, textboxes, controls, frame windows, links, list boxes, tables, combination(combo) boxes, tabs, sliders, etc. Graphical or GUI objects may havecorresponding instantiated or software objects. In some embodiments,objects in the programming or software sense and according to theclass-based object-oriented programming model may be not actualinstantiated object-oriented objects, but rather a construct usedinternally by an executing application to represent on-screen objects.

Some embodiments of the invention use for example two phases or modes: adesign-time phase or mode for defining and collecting graphical shapesinformation, and a run-time phase or mode for applying the graphicalshapes information to an application in real time. When referred toherein, design-time may refer to an analysis, preparatory or setup typemode or stage during which embodiments of the invention may analyze anapplication image (which may vary over time as the application statechanges or updates, and which thus may require that the application isexecuted over a period of time), e.g., by applying one or more imageprocessing algorithms, to learn about, identify, define, and/or mark-upobjects in the application image such that the those objects can belater identified, constructed, and/or made operable during a run-timemode or stage. In some embodiments, during a design-time mode, a sceneand/or control anchors (e.g., shape anchors) may be defined, geometricrelationships between anchors may be identified, GUI control types maybe specifying, and controls data, anchors data and/or the applicationimage may be collected and/or stored.

Run-time may refer to mode or stage during which the application isactually running and available for operability. As applied to someembodiments of the invention, during a run-time mode, an appropriatescene (e.g., the application state) may be recognized, for example,based on the collected design-time data, and interaction with GUIelements/objects of application may be enabled, e.g., as though a userwere interacting directly with the application.

In some embodiments, as described in detail herein, during a design-timemode, actions/steps may be implemented such as for example: edges andcontours detection; polygonal approximation of each contour; calculationof shape bounding rectangle(s); defining shape properties (e.g., shapeanchors and/or shape controls), such as approximate points, childrencounts, rectangles, child contours, etc.; defining scene anchors forcontrol shapes. defining geometric relationships between anchors toidentify controls of the scene at run-time; specifying GUI control types(e.g., shape control types), such as frame windows, buttons, links, listboxes, tables, combo boxes, tabs, sliders, etc.; and calculating animage histogram for each shape (e.g., to compare shape histogramsinstead of image comparison). Different and/or other actions may beused.

In some embodiments, during a run-time mode, actions/steps may beimplemented such as for example: recognizing (e.g., identifying,finding, etc.) the frame window shape, e.g., by comparison of the imagehistogram (calculated at design-time) with an image histogram (createdat run-time) or an application image captured at run-time; identifyingone or more shape anchors by comparison of design-time/run-timehistograms and/or the maximum coincidence of child contours; identifyingcontrol shapes by comparison of design-time/run-time histograms andgeometric relationships between control anchors; intercepting imagepaint calls (e.g., BitBlt, StrechBlt, etc.) to identify image changes orupdates(representing, e.g., various application states) and provideperformance optimization for remote control applications; recognizinginput device (e.g. mouse/touchscreen and keyboard) hooks (e.g., via theWindows API) indicating user activity events; recognizing graphicsdevice interface (GDI) drawing functions occurring on the clientcomputer or terminal; recognizing Windows or other GUI system eventshook and subclass window procedures, e.g., redraw messages; recognizinga timer timeout or the end of a time period; and emulating run-timecontrols and their various actions, properties, and events, etc.Different and/or other actions may be used.

FIG. 1 shows a high level diagram illustrating an example configurationof a system 100 for enabling interoperability with a run-timeapplication (or simply an application), according to at least oneembodiment of the invention. System 100 includes network 105, which mayinclude the Internet, one or more telephony networks, one or morenetwork segments including local area networks (LAN) and wide areanetworks (WAN), one or more wireless networks, or a combination thereof.System 100 includes a system server 110. In some embodiments, systemserver 110 (also referred to as a remote server) may be a stand-alonecomputer system. In other embodiments, system server 110 may include anetwork of operatively connected computing devices, which communicateover network 105. System server 110 may include multiple processingmachines such as computers, and more specifically, stationary devices,mobile devices, terminals, and/or computer servers (collectively,“computing devices”). Communication with these computing devices may be,for example, direct or indirect through further machines that areaccessible to the network 105.

System server 110 may be any suitable computing device and/or dataprocessing apparatus capable of communicating with computing devices,other remote devices or computing networks, receiving, transmitting andstoring electronic information and processing requests as furtherdescribed herein. System server 110 is therefore intended to representvarious forms of digital computers, such as laptops, desktops,workstations, personal digital assistants, servers, blade servers,mainframes, and other appropriate computers and/or networked or cloudbased computing systems capable of employing the systems and methodsdescribed herein.

System server 110 may include a server processor 115 which isoperatively connected to various hardware and software of system 100.Server processor 115 serves to execute instructions to perform variousoperations relating to embodiments of the invention. Server processor115 may be one or a number of computer processors, a central processingunit (CPU), a graphics processing unit (GPU), a multi-processor core, orany other type of processor.

System server 110 may be configured to communicate via communicationinterface 120 with various other devices connected to network 105.Server memory 125 may be accessible by server processor 115, therebyenabling server processor 115 to receive and execute instructions such acode, stored in the memory and/or storage in the form of one or moresoftware modules 130, each module representing one or more code sets.Software modules 130 may include one or more software programs orapplications (collectively referred to as the “server application”)having computer program code or a set of instructions executed partiallyor entirely in server processor 115 for carrying out operations foraspects of the systems and methods disclosed herein, and may be writtenin any combination of one or more programming languages. Serverprocessor 115 may be configured to carry out embodiments of the presentinvention by, for example, executing code or software, and may executethe functionality of the modules as described herein.

Server modules 130 may include more or less actual modules which may beexecuted to enable functionalities of the invention. Server modules 130may be executed entirely on system server 110 as a stand-alone softwarepackage, partly on system server 110 and partly on user device 140, orentirely on user device 140. Device 140 (also referred to as a localcomputer) may be remote from third party server 180, also referred to asa remote computer.

Server memory 125 may be, for example, a random access memory (RAM) orany other suitable volatile or non-volatile computer readable storagemedium. Server memory 125 may also include storage which may takevarious forms. For example, the storage may contain one or morecomponents or devices such as a hard drive, a flash memory, a rewritableoptical disk, a rewritable magnetic tape, or some combination of theabove. In addition, the memory and/or storage may be fixed or removable.In addition, memory and/or storage may be local to the system server 110or located remotely.

System server 110 may be connected to one or more database(s) 135, forexample, directly or remotely via network 105. Database 135 may includeany of the memory configurations as described herein, and may be indirect or indirect communication with system server 110. In someembodiments, database 135 may store information relating to userdocuments. In some embodiments, database 135 may store informationrelated to one or more aspects of the invention.

User device 140 may connected to the network 105 and may be any standardcomputing device, for example a desktop computer, smart terminal, dumbterminal, kiosk and/or other machine, each of which generally has one ormore processors, such as user processor 145 (also referred to as localprocessor), configured to execute code, a computer-readable memory, suchas user memory 155, a user communication interface 150, for connectingto the network 105, one or more user modules 160, one or more inputdevices 165, and one or more output devices 170. Typical input devices,such as, for example, input devices 165, may include a keyboard,pointing device (e.g., mouse or digitized stylus), a web-camera, and/ora touch-sensitive display, etc. Typical output devices, such as, forexample output device 170 may include one or more of a monitor, a localdisplay, speaker, printer, etc.

In some embodiments, user module 160 may be executed by user processor145 to provide the various functionalities of user device 140. Inparticular, in some embodiments, user module 160 may provide a userinterface with which a user of user device 140 may interact, to, amongother things, communicate with system server 110. For example, systemserver 110 may execute a client/server, “target” or “monitored”application 142 as the server of a server/client architecture, and userdevice 140 may display the GUI display as controlled by server 110 andaccept input to send to server 110.

In some embodiments, user device 140 may be or act as a “dummy”terminal, by which processing and computing may be performed on systemserver 110, and information may then be provided to user device 140 viaserver communication interface 120 for display and/or basic datamanipulation. In some embodiments, modules depicted as existing onand/or executing on one device may additionally or alternatively existon and/or execute on another device. For example, in some embodiments,one or more modules of server module 130, which is depicted in FIG. 1 asexisting and executing on system server 110, may additionally oralternatively exist and/or execute on user device 140. Likewise, in someembodiments, one or more modules of user module 160, which is depictedin FIG. 1 as existing and executing on user device 140, may additionallyor alternatively exist and/or execute on system server 110.

A computing device discussed herein may be a mobile electronic device(“MED”), which is generally understood in the art as having hardwarecomponents as in the stationary device described above, and beingcapable of embodying the systems and/or methods described herein, butwhich may further include componentry such as wireless communicationscircuitry, etc. Non-limiting examples of typical MEDs are smartphones,personal digital assistants, tablet computers, and the like.

Third party server 180 may operate software or applications which mayinteract with or receive information from target or monitoredapplication 142 operated via system server 110 and displayed on userdevice 140 via methods as disclosed herein. Since third party server 180may not have access to, or API access to, client software operated bysystem server 110 and displayed in a GUI on device 140, embodiments ofthe invention may provide an API for third party server 180 to interactwith the client software via a specially made GUI which gathersinformation from graphical changes on user device 140. For example,system server 110 may execute a client/server, “target” or “monitored”application 142 as the server of a server/client architecture, and userdevice 140 may display the GUI display as controlled by server 110 andaccept input to send to server 110.

For example, third party server 180 may operate software or applicationssuch as RT server software modules 188 (e.g. the Real-Time ProcessOptimization available from NICE, of Raanana, Israel) which are intendedto monitor and possibly control an interaction between a customer and ahuman agent, where the human agent is using application 142: thisinteraction between RT server 188 and agent application 142 may beperformed by embedded or monitoring software typically executed on thesame computer that executes agent application 142 such as RT client 144.Software such as RT server software modules 188 may communicate withagent application 142 and may be remote from agent application 142, andthis communication may be via RT client 144. Third party server 180 mayreceive information from application 142 (which may be considered amonitored application) and in some cases may send messages to or controlapplication 142, for example send messages including suggestions to theagent regarding the customer interaction. One or more items of embeddedsoftware, monitoring software or RT client 144 may be executed by userdevice 140 to monitor and/or communicate with application 142 andgenerate events, alerts, etc., and to operate or act as software objectsto allow interaction between third party server 180 and application 142.Third party server 180 may include one or more input devices and outputdevices such as, for example, keyboards, pointing devices, monitors,displays, speaker, printer, etc. In one embodiment RT server 188 orother software interacts with agent application 142 via embeddedsoftware such as RT client 144. In another embodiment agent application142 is controlled by and sends input to RT client 144 with no otherprogram controlling RT client 144. Generating, triggering or raising anevent may include notifying a process (e.g., a process within or part ofRT client 144, or a process external to RT client 144, such as RT serversoftware modules 188, of the change.

Third party server 180 may communicate over network 105 and may includeone or more processing machines such as computers. Third party server180 may be any suitable computing device and/or data processingapparatus such as servers, laptops, desktops, workstations, personaldigital assistants, etc. Third party server 180 may include one or morecomputer processors 185 which may be configured to carry out methods asdisclosed herein (possibly in conjunction with processors 115 and 145)and may be one or a number of computer processors, a central processingunit (CPU), a graphics processing unit (GPU), a multi-processor core, orany other type of processor. Third party server 180 may include memory182 storing for example data and/or code such as RT server 188. RTserver 188 may include one or more software programs or applicationshaving computer program code or a set of instructions executed partiallyor entirely by processor 185 for carrying out operations as disclosedherein. Modules such as RT server 188 may for example communicate withRT client 144 to receive input from and/or control or send instructionsto monitored application 142, or cause application 142 to receive input,such as text inserted into a textbox, or controls selected or clicked,at the initiative and control of modules such as RT server 188. Suchcommunication may be performed by software objects corresponding tocontrols within application 142. Processor 185 may be configured tocarry out embodiments of the present invention by, for example,executing code or software, and may execute the functionality of RTserver 188 or other modules.

Third party server memory 182 may be, for example, a random accessmemory (RAM) or any other suitable volatile or non-volatile computerreadable storage medium and may include storage which may take variousforms, and may be located remotely.

FIG. 2 is a flow diagram of one embodiment for a “designer” stage, foridentifying and/or defining GUI objects in an application image during adesign-time learning stage, mode, or process, according to embodimentsof the invention. FIG. 2 shows a learning method or process in whichcharacteristic data models or scenes are generated and stored for laterrecall. Furthermore, in some embodiments, method 200 may be configuredto implement one or more of the elements/features/functions of system100.

As with other methods described herein, method 200 may be performed on acomputer having a processor, a memory, and one or more code sets storedin the memory and executed by the processor, such as but not limited tothe devices depicted in FIG. 1. At step 205 a design-time applicationimage is received. Receiving may include, for example, capturing theimage of the application as a “screenshot,” e.g., by use of aprint-screen function or other image and/or screen capture method and/ordevice, or receiving a previously captured image of the application. Insome embodiments, the application image may include, for example, anentire application window, a portion of an application window, an entiredisplay including portions and/or entire windows of one or moreapplications, a portion of a display, etc. For example, turning brieflyto FIG. 3, a captured application image 300 is shown according to atleast one embodiment of the invention.

At step 210, the captured/received application image may be transformedor converted to a greyscale version of the image. For example, the imagemay be processed with a grayscaling function (CvCvtColor) from OpenCVlibrary, or the like. For example, turning briefly to FIG. 4, a capturedapplication image 400 is shown, with edges and contours, converted tograyscale. In some embodiments, transforming or converting theapplication image to grayscale may not be performed, for example, inembodiments where color does not impact detection/identification ofedges and/or contours in the application image and/or in embodimentswhen the application image is already in grayscale.

At step 215, one or more (e.g., two) threshold values may be definedwhich may impact detection of edges and contours in an image. Forexample, in some embodiments of the invention, the processor may receiveone or more threshold values from a user to be implemented in an edgedetection algorithm such as the Canny operator or the Canny EdgeDetector function (CvCanny) from OpenCV library, or the like. In someembodiments, the processor may be configured to automatically determineoptimal or near optimal threshold values for detection of edges.

The Canny edge detector is an edge detection operator that uses amulti-stage algorithm to detect a wide range of edges in images. Cannyedge detection is a technique to extract useful structural informationfrom different visual objects (e.g., objects visually represented in anapplication image) and dramatically reduce the amount of data to beprocessed. In some embodiments, the Canny edge detector may apply aGaussian filter to smooth the image in order to remove noise, find theintensity gradients of the image, apply non-maximum suppression toremove spurious responses to edge detection, track edges by hysteresis,and/or finalize the detection of edges by suppressing edges that areweak and not connected to strong edges, etc.

At step 220, one or more edges and/or one or more contours in theapplication image may be found and/or identified. For example, in someembodiments, the processor may be configured to identify one or moreedges, e.g., by executing an edge detection algorithm such as the CannyEdge Detector function (CvCanny) from OpenCV library, or the like.Furthermore, for example, in some embodiments, the processor may beconfigured to identify one or more contours, e.g., by executing acontour detection algorithm such as the CvFindContours function fromOpenCV library, or the like. Such a contour detection algorithm may findcontours using edges returned from an executed edge detection algorithm(e.g., CvCanny function). Of course, those of ordinary skill in therelevant art will understand that there are a number of algorithms whichmay be implemented, alone or in combination, to filter, identify, and/ordetect edges (Edges detectors), for example: Canny edge detector(operator), Sobel operator, Laplace operator, etc.

In some embodiments, for example when the application image is quitecomplex, visual objects in the application image may be distinguished byidentifying contours (and/or edges) of such visual objects within theapplication image. As understood herein, a contour may be defined as acurve joining a plurality of continuous points (e.g., along theboundary), having the same color or intensity. A contour may be, forexample, an external outline (e.g., stroke) of a visual object thatseparates it from the background and/or other visual objects in theimage. Algorithms known to those of ordinary skill in the art and/or asdescribed herein may be executed in various embodiments to implementconvenient methods for the detection and manipulation of image contours.For example, a FindContours function of the OpenCV library may be usedfor retrieving, detecting, and/or identifying contours. In someembodiments, a processor may implement an approximation method which maycompress one or more horizontal, vertical, and/or diagonal segments,leaving, e.g., only their end points (for example, using aCV_CHAIN_APPROX_SIMPLE method). Detection of edges and contours in anapplication image enables defining of bounding shape objects as resultof this processing, as described herein.

At step 225, one or more shape size values may be defined. A shape sizevalue may define, for example, a minimum and/or maximum perimeter (e.g.,shape size) for which a bounding shape object may be generated. Abounding shape object may be defined as a bounding shape (e.g., arectangle), for example, a minimum bounding shape, surrounding,bounding, enclosing, identifying, and/or otherwise relating to aspecific or given contour, set of contours, set of one or more edgesand/or one or more contours (e.g., typically an identified visual objector other shape), etc. As such, one or more shape size values may bedefined in accordance with embodiments of the invention to define, e.g.,minimum and/or maximum acceptable sizes for bounding shape objects to begenerated as explained herein. Minimum and/or maximum shape values may,for example, prevent the system from bounding unnecessary and/orirrelevant visual objects (e.g., visual objects which are likely toosmall or too large to be GUI control elements or objects of interest tothe user, but which nonetheless have a definable contour). In someembodiments, one or more shape values may be calculated, estimated,recommended, and/or suggested automatically by the processor. In someembodiments, the processor may receive one or more shape values as inputfrom a user. Furthermore, in some embodiments, e.g., when too manyvisual objects are identified, the processor may be configured to removevisual objects/shapes with bounding rectangles less than a previouslydefined shape size, e.g., in real-time, for example, based on feedbackfrom a user.

At step 230 a bounding shape object (e.g., a bounding shape such as aminimum bounding rectangle) for one or more contours, one or more edges,and/or one or more visual objects identified in the application imagemay be created and/or generated. For example, FIG. 5 depicts a set ofgenerated bounding shape objects 500 is shown according to at least oneembodiment of the invention. In should be noted that while in theexample embodiment of FIG. 5 rectangles were generated to bound thevarious objects, in other embodiments other regular and/or irregularshapes may also or alternatively be generated in order to define theboundaries of various visual objects, shapes, contours, and edges withinan application image as appropriate.

At step 235 a shapes tree (e.g., a shape object tree) based on thevarious bounding shape objects generated in step 230 may be built. Insome embodiments, for example, all bounding shape objects with abounding rectangle larger than e.g., a defined minimum rectangle sizemay be placed in a shapes array. This array may be processed, and ashape objects tree may be built. For example, FIG. 6 depicts an exampleportion of a shapes tree 600 according to at least one embodiment of theinvention.

A hierarchy of the shapes tree may be defined using an algorithm based,e.g., on coordinates nesting. Each tree node (e.g., representing abounding shape object) may have an associated set of properties, forexample: path in tree, coordinates (e.g., relative to the top leftcorner of the image), histogram data (as described herein), text if any(e.g., recognized using OCR), child (e.g., internal) shapes andcontours, etc. In some embodiments, the shapes tree may be built inaccordance with nesting coordinates. For example, each shape whichencloses one or more other shapes may be considered as a parent and allshapes enclosed within may be considered as children.

In order to include histogram data in the shapes tree, in someembodiments a histogram (e.g., a design-time histogram) may beconstructed based on the defined bounding shape properties of thevarious bounding shape objects. Histograms can be used to represent suchdiverse information as the color distribution of an object, an edgegradient template of an object, the distribution of probabilitiesrepresenting an expected object location, etc. Therefore, in someembodiments, one or more points of interest may be identified in anapplication image by assigning each point of interest a “tag” consistingof histograms of nearby features. Histograms of edges, colors, cornersand so on may form a general feature type that is passed to classifiersfor object recognition. In some embodiments a first histogram may begenerated during a design-time mode, and a second histogram may begenerated during a run-time mode, at which time the histograms may becompared to enable such object recognition, e.g., based on similarity ofhistogram data.

At steps 240-255, one or more bounding shape properties for one or moreof the bounding shapes (e.g., bounding shape objects) may be defined,identified, associated, received, and/or selected. Such properties mayinclude, for example, defining anchors and controls, etc. At step 240,at least one anchor (e.g., a shape anchor) may be defined, identified,associated, received, and/or selected from among the one or morebounding shape objects. An anchor may represent a visual object in thedesign-time application image having content or an image which remainsconstant from the design-time mode to the run-time mode. Each anchor maydenote shape data selected to be used in the scene identification atrun-time. A set of anchors uniquely determines a scene. An anchor may befor example an object based on a shape of which internal (e.g., child)content (e.g., contours) is constant and cannot be changed fromdesign-time to run-time. For example, a shape of button with constanttext or an image on the face of the button may be defined as an anchoras the features of the button are not expected to change fromdesign-time to run-time and can therefore be used to recognize theapplication as part of a scene. In some embodiments, one or more anchorsmay be identified, determined, recommended, and/or suggestedautomatically by the processor. In some embodiments, a processor mayreceive one or more anchor selections as input from a user.

At step 245, at least one control, also referred as control object,(e.g., a GUI control) may be defined, identified, associated, received,and/or selected from among the one or more bounding shape objects. Acontrol object may represent a controllable object, e.g., an object thatthe user of the application may manipulate during run-time mode, locatedwithin the design-time and the run-time application images. A controlmay associate a given object with a control type for which aninteraction with the control is defined within the run-time stage. Forexample, a control type may include one of a frame window, a button, aradio button, a link, a list box, a check box, a table, a drop-list, acombination box, a tab, and a slider, etc. Manipulating a control objectmay include performing an operation on the control, typically throughthe user interface (e.g., the GUI) using for example a mouse or akeyboard, including for example changing text, pressing a button, etc.The specific manipulation of a control may depend on the control type.For example, manipulating a button control may include selecting orpressing the button, manipulating a table may include scrolling throughthe table and/or changing data in a table box, etc. In some embodiments,one or more controls may be identified, determined, recommended, and/orsuggested automatically by the processor. In some embodiments, theprocessor may receive one or more control selections as input from auser. Furthermore, in some embodiments, one or more control types forselected controls may be identified, determined, recommended, and/orsuggested automatically by the processor. In some embodiments, theprocessor may receive one or more control type selections as input froma user.

At step 250, one or more bounding shape objects (e.g., additional shapeobjects) may be created on the basis of a shape (e.g., a rectangle)drawn (e.g., manually marked and/or otherwise digitally added) on theimage. Furthermore, the newly created shape object may be added to theshape objects tree.

At step 255, in some embodiments, one or more controls may be attachedor otherwise associated to one or more anchors. By attaching controls todefined anchors, one or more geometric relationships may be definedbetween controls and anchors, the data of which may be used to identifycontrols of a scene at run-time, as described herein. In someembodiments, one or more controls may be attached to one or more anchorsautomatically by a processor. In some embodiments, a processor mayreceive attachment selections as input from a user. Furthermore, in someembodiments, one or more geometric relationships between controls andanchors may be calculated identified, determined, recommended, and/orsuggested automatically. In some embodiments, a processor may receiveone or more geometric relationship calculations or selections as inputfrom a user.

At step 260, data relating to one or more of the design-time applicationimage, the one or more design-time visual objects, the one or moredesign-time bounding shape objects, the one or more bounding shapeproperties, the shape tree data, and/or the design-time histogram datamay be stored, e.g., as a scene or another data object, for laterrecall. A scene may be for example a project object which represents anapplication state. It may include, for example, the captured applicationimage, a set of anchors and/or controls, etc., e.g., defined by a usersuch as a project designer. A scene may provide one or more objectmodels which may be later recalled and used during run-time to identifyan application and enable the various defined controls for theapplication. An object model, as understood herein, may be defined as aset of properties (e.g. data stored by the control), methods (e.g.software functions that an external program may cause the control toexecute), features, and/or events (e.g. software functions may be raisedor fired, or the equivalent of raising or firing may be performedinternal to a program, to produce output to an external program from thecontrol) specific for an identified or given object. FIG. 7 depicts anexample scene object model 700 according an embodiment of the invention.Furthermore, as shown in FIG. 8, various scene data may be provided,recorded, stored, etc., in a scene design-time data table 800 as shownaccording an embodiment of the invention. In some embodiments,additional scenes may be created, e.g., by capturing and processingadditional application images as necessary. Otherwise, the design-timemode may end.

To provide reliable interaction and operability with an application itmay be necessary to identify the application state, particularly openwindows, visible GUI elements, etc. Accordingly, embodiments of theinvention enable a processor to identify the relevant application statebased on the previously stored scene (e.g., scene object model and/orscene table data, etc.). FIG. 9 is a flow diagram a method for enablinginteroperability with a run-time application according to an embodiment.In particular, FIG. 9 depicts a flow diagram of a method 900 foridentifying an application based on detected visual elements (e.g., GUIobjects or control graphic items) of the application during a run-timestage, mode, or process, according to an embodiment. In someembodiments, method 900 may be configured to implement one or more ofthe elements/features/functions of system 100, or another system.

At step 905 a scene (e.g., the scene stored in step 260 of FIG. 2)containing design-time data may be received, recalled, read, and/orreloaded. In some embodiments, while a client application is starting up(e.g., loading), for example, on a remote desktop environment on aclient/user device, the scene may be concurrently recalled and thedesign-time scene data (e.g., design-time (DT) application image,anchors and controls with DT properties, threshold values, shape sizesetc.) may be read and/or loaded. Of course, in other embodiments, scenedata may be preloaded even before an application is run.

At step 910, one or more design-time bounding shape objects forimplementation or use during run-time may be created (e.g., recreated).In some embodiments, one or more of bounding shape objects, acorresponding shapes tree, bounding shape properties, anchors, controls,etc., may be built, generated, constructed, and/or calculated with thesame or similar algorithms, processes, methods, etc., as at design-time,described in detail with regard to FIG. 2. In some embodiments, for eachanchor and/or control defined in the scene during the design-time mode,a sub-image may be extracted from the design-time application image. Insome embodiments, the extraction may be based, for example, on relativecoordinates of the shape object with respect to an anchor or a control.Such sub-images may be used for example, to overlay and/or identifycontrols and/or anchors in an application during run-time, e.g., inplace of bounding objects. In some embodiments, while rebuilding aproject (e.g., a scene and/or one or more components of the scene) andapplying it at run-time, the application image may therefore be splitinto a set of pieces (e.g., saved as sub-images) corresponding to thevarious GUI element recognized. For example, FIG. 7 depicts a List Box,Button Control, Shape Anchor, Frame Window Control, etc., which may beextracted as sub-images, which may be used as overlays during run-timecreation of bounding shape objects.

At step 915, the DT Histogram may be calculated (e.g., recalculated)(e.g., as previously described in step 235). In some embodiments, the DThistogram may be calculated using a sequential call to a library suchas, e.g., OpenCV to execute functions (e.g., CvCreateHist, CvCalcHist,CvNormalizeHist) to create, calculate, and normalize the histogram.

At step 920 a run-time (RT) application image may be captured, and insome embodiments may be saved or cached. Such capturing may be caused orinitiated by any of one or more change triggers, depending on theembodiment, such as a timeout or expiry of a period of time, or adetected screen change or paint command, or a user input that has thepotential to change the screen. As understood herein, capturing mayinclude, for example, capturing the image of the application as a“screenshot,” e.g., by use of a print-screen function or other imageand/or screen capture method and/or device, or receiving a capturedimage of the application, e.g., from an external image capturing device,etc.

Other or different operations may take place. In other embodiments, aprocessor or a process may implement different methods and/or processesto take or capture the application image at run-time, depending on theapplication type and/or the environment in which it is running. Forexample, in some applications (such as the Remote Desktop application,for example) which are drawn on the screen only by image functions, theprocessor may be configured to intercept image paint calls (e.g.,BitBlt, StrechBlt, etc.). For other applications, in some embodiments,the processor may capture a screenshot when an application repaint eventarrives. In some embodiments, the RT application image may include, forexample, an entire application window, a portion of an applicationwindow, an entire display including portions and/or entire windows ofone or more applications, a portion of a display, etc.

At step 925 one or more run-time bounding shape objects (e.g.,instantiated objects corresponding to graphically displayed controls, orinternal representations corresponding to instantiated objects) may becreated, for example, corresponding to, relating to, and/or associatedwith, one or more design-time objects, as described herein. Inparticular, in some embodiments, one or more RT Image bounding shapeobjects and/or a run-time shapes tree may be created, built,constructed, generated, and/or calculated with the same or similaralgorithms, processes, methods, etc., as at design-time, described indetail with regard to FIG. 2, e.g., on the basis of one or moreidentified bounding shape properties as described herein. In someembodiments, the run-time state of the application may not coincideprecisely or at all with the design-time state due to resizing and/orhow the application appears on the screen in other environments, inother applications (windows), etc., and therefore the set of boundingshape objects and/or the shape tree may, in some embodiments, beslightly or substantially different than those generated/created duringdesign-time. As such, in some embodiments, for each anchor and/orcontrol, the processor may be configured to search for appropriateshapes of the suitable size and to detect tree branch overlapped shapes.In some embodiments, one or more (e.g., typically each) of the RTbounding shape objects may be generated and/or created as an appropriatetype so as to expose elements and features of the scene object model,e.g., specific for each object type. In some embodiments, these RTbounding shape objects may be overlaid with corresponding sub-images soas to replicate or otherwise visually represent the correct object typeof the GUI elements for which they were created.

At step 930, in order to identify the proper scene to associate with therun-time application, an attempt may be made to identify the framewindow control of the run-time application, e.g., presuming it wasdefined in the scene at design-time. If no application frame window canbe identified, at step 935, in some embodiments, just the run-timewindow application may be captured, e.g., via a screenshot, e.g., todetect the necessary frame window information. If a frame window controlis identified, then at step 940, one or more frame window shapes in theRT application image may be searched for, detected, found, and/oridentified. In some embodiments, the frame window control in the RTapplication image may be searched for, possibly via its associated data,e.g., by executing one or more histograms comparison functions of DTframe window control data (e.g., collected during design-time) and RTshape histograms (e.g., using the OpenCV function CvCompareHist). Thecomparison result may be the array of RT shape objects whose histogramvalues are similar.

At step 945, one or more RT anchor shapes (e.g., anchors) may be found(e.g., detected, identified). To find RT anchors in the entire scene(e.g., in the scene model and/or in the scene data) or a portionthereof, in some embodiments, a method may again execute one or morehistogram comparison functions, as described above. In some embodiments,a processor may compare design-time object data to image data collectedat run-time, e.g., with or without comparing histograms, depending onthe instance, e.g., by comparing contours. In some embodiments, e.g., ifthe contours are essentially identical, a processor may continueverification of the scene.

Anchors may contain information/data regarding contours calculated forthe shape content (picture, text, lines, etc.) that are typicallyconstant; an anchor may be for example a program or application name, alogo, etc. As such, detection of matching anchors is typically anindication that a correct (matching) scene has been identified. At step950, if no anchors are detected, then, the run-time application imagemay be recaptured at step 955, for example during an application repaintevent or on the occurrence of another change trigger, and the processmay continue with step 925. In some embodiments, recapturing theapplication may resolve any mismatching and/or detection issues.

If one or more anchors are found at step 945, then at step 960, one ormore control shapes (e.g., controls), may be found find (e.g., detected,identified) for example, based on identified anchors. In someembodiments, controls may be searched for in the same or a similarmanner as anchors. To find RT controls in the entire scene (e.g., in thescene model and/or in the scene data) or a portion thereof, in someembodiments one or more histogram comparison functions may be executedagain, as described above. In some embodiments, a processor may comparedesign-time object data to image data collected at run-time, e.g., withor without comparing histograms, depending on the instance. However, itshould be noted that controls are not necessarily constant (as theytypically are with anchors), and their contours may be somewhatdifferent, and thus controls may, in some embodiments, be more readilyidentified via comparison of histograms.

At step 965, if no controls are detected, then the run-time applicationimage may be recaptured at step 955, for example during an applicationrepaint event, and the process continues with step 925. In someembodiments, recapturing the application may resolve any mismatchingand/or detection issues.

At step 970, one or more control shapes (e.g., controls) may be found(e.g., detected, identified, etc.), for example, based on previouslydefined or identified geometric relationships. To find RT controls inthe entire scene (e.g., in the scene model and/or in the scene data) ora portion thereof, in some embodiments, for example, when one or morecontrols have been bound or attached to a particular anchor or anchors,a processor may identify such controls based on searching geometricrelationships between controls and anchors. In some embodiments, such aswhen one or more controls have not been bound or attached to anyspecific anchors, a processor may search for and detect geometricrelationships between controls and any or all anchors already identifiedon the scene. Of course, the processor may execute one or more histogramcomparison functions as well, as described above.

At step 975, if again no controls are detected, the run-time applicationimage may be recaptured at step 955, for example during an applicationrepaint event, and the process may continue with step 925. In someembodiments, recapturing the application may resolve any mismatchingand/or detection issues.

At step 980, in some embodiments any user-drawn controls (e.g., controlshapes) may be found (e.g., detect, identify, etc.). To recognizeuser-drawn control shapes and their respective coordinates, in someembodiments, a processor may calculate geometric relationships betweenthe specific control and any or all other controls and anchors found onthis scene.

At step 985, e.g., when all or a threshold amount of the anchors andcontrols defined in the scene are recognized, the scene may be marked asrecognized (e.g., identified) and interaction with the GUI of therun-time application may be enabled. For example, in some embodiments,in order to realize the object model, a processor may emulate run-timecontrol methods, properties and/or events of the design-time boundingshape objects by, e.g., generating, constructing, assigning, defining,or otherwise enabling the one or more RT bounding shape objects (e.g.,instantiated objects corresponding to graphically displayed controls orinternal constructs using as a model such objects such as objectconstructs) such that they have the same or a similar defined set ofproperties, methods, features, and/or events specific for an identifiedor given object as those of the one or more DT bounding shape objects.For example, for input emulation (mouse, keyboard, etc.) a processor mayuse such functions as SendInput, SendMessage and PostMessage functions(e.g., for the Windows API). The processor may further use mouse andkeyboard hooks (Windows API) to receive user activity events. Forcontent change events a processor may check and identify contourschanging in the control. As before, a processor may also intercept imagepaint calls (BitBlt, StrechBlt etc.) in order to change the applicationstate. Turning briefly to FIG. 10, a table 1000 of example controlmethods, properties, and events is provided according to an embodimentof the invention.

FIG. 11 is a flow diagram of one embodiment for a “designer” stage (e.g.application 146 executed for example by third party server 180 oranother device), for identifying and/or defining GUI objects in anapplication image (e.g. a client/server or monitored application such asapplication 142 executed on a user terminal) during a design-timelearning stage, according to embodiments of the invention. Aspects ofFIG. 2 may be used in conjunction with FIG. 11, and portions of the twoembodiments may be combined. Application 142, shown in FIG. 1 as beingexecuted on a client system, may also be executed on a third partysystem, for the purposes of making its image available to an RT designapplication, and for the purposes of design of RT client 144.

In operation 1100, two processes may be opened or executed on a userterminal such as third party server 180: the application to be analyzed(e.g., application 142 executed during a designer phase), and a RTdesign tool or application 146 operated by a designer to define forexample: graphical objects or controls displayed by application 142 andto be monitored, and software objects which are to represent thegraphical entities (objects, controls, etc.) of application 142. In oneembodiment, the two applications may be operated side by side by thesame processor, but in other embodiments the applications may beexecuted remotely from each other.

In operation 1110 a user may operate application 142 to operate, alteror select a control or other control graphic item displayed by the GUIof application 142, or select another GUI element of application 142.For example, a user may use input devices associated with a computerwhich is part of third party server 180 and click on or select acontrol, text box item, list item, etc. A user may cause a GUI elementof application 142 to be altered, for example by typing text whichappears in and thus changes or updates the appearance of a text box. Auser may essentially operate application 142 to have all controls deemedrelevant by the user operated and thus analyzed and defined, as per theoperations below. A user may highlight or select another GUI element ofapplication 142: for example a user may select (e.g. using a selectionbox or tool) an anchor of application 142.

In operation 1120, with RT design application 146, the control or GUIelement operated or altered in operation 1110 may be recognized, or itsexpected location may be recognized, for example using tools or methodsdiscussed with respect to FIG. 2, or other methods. A display inapplication 146 mirroring the GUI displayed by application 142 mayindicate in highlight (e.g., a colored border) the control or GUIelement operated or altered by the user in application 142.

In operation 1130, with RT design application 146, attributes may beassigned to the highlighted or recognized control operated or altered,possibly with user input. For example, a user may operate application146 to define a recognized control as a text box, button etc., and/ormay define or select a software object (e.g. as shown in FIG. 10) tocorrespond to the recognized control, and a user may define events,methods, and other properties for the software object. While operation1130 may be performed with the aid of user input, all or some ofoperation 1130 may be performed automatically, by a processor.

In operation 1140 RT design application 146 software objects may becreated or defined to be later simulated or instantiated during run time(e.g. as shown in FIG. 10) corresponding to the controls or GUI items,or may create other data corresponding to the GUI items. Such softwareobjects may be object-oriented programming software objects. Dataincluded in or associated with such objects may include for example animage (e.g. a graphical representation) of the control which may be usedas a comparison to a run-time image of the control, and other data. If anon-control element (e.g., anchor) is operated or identified inoperation 1110, it may not be the case that a software object isassigned to the element. For example, the element may be recognized forthe purpose of identifying application 142 or the layout of application142. An anchor may be used during run time for recognizing anapplication, recognizing a position of an application, and defining theexpected other GUI elements or controls, defined by the data created atdesign time as existing spatially relative to the anchor. RT designapplication 146 may also accept user input defining software objects andmay, during the design phase, provide input to controls or otherelements of application 142 and view events or other changes or outputoccurring with respect to software objects associated with thosecontrols.

In operation 1150 RT design application 146 may save the data created,for example as a “scene.” Such data may define or control RT client 144,when it is executed. For example a scene may be compiled into a DLL usedby an RT client application. In some embodiments, data created duringthe design phase may define how RT client 144 operates during run time,or may define or be RT client 144 itself.

Embedded or other software (e.g. embedded software or application suchas RT client 144) may monitor the state of a GUI intensive clientapplication, e.g. application 142, when the application is running. Forexample, RT server 188 may communicate with RT client 144 to receiveinput from and/or control or send instructions to monitored application142, or cause application 142 to receive input, such as text insertedinto a textbox, or controls selected or clicked, via software objectscorresponding to controls within application 142.

Other or different operations may be used.

After design-time, when a client application (e.g. application 142), theapplication may be run at “run time” using the design-time definitionsto allow third-party software to interact with the client applicationwithout direct interaction or access to the server operating the client.In order to do this, during run-time, changes or updates in the GUIdisplay or graphical image of the client application over time may bedetected. The determination of whether there is a change to detect torelevant GUI controls or other GUI aspects of interest to thethird-party software may first be made on the occurrence of a changetrigger, an event or message which may signal the possibility that theclient image has changed in a way relevant to a control or other change.Such a change trigger may be for example one or more of a timeout orexpiry of a period of time, a detected screen change or paint command(e.g. from the OS or Windows level), and a user input that has thepotential to change the screen. Other suitable change triggers may beused.

For example, a first image of the display (e.g. of target, client ormonitored application 142) may be stored or cached (e.g. by RT client144), and at some point, e.g., after a change trigger, a second imagemay be stored or cached, the first image in this context being a priorimage and the second image being the current image. This capturing maybe iterative, where the current image becomes the prior image. Based onrelevant changes or updates (e.g., by detected by RT client 144), aprocess on a client system (e.g. user device 140) communicate changes instate or messages to a third-party system, simulating a GUI. Forexample, events such as TextChanged may fire in response to detectedchanges between a first and second image. “Firing” or “raising” an eventmay in some embodiments be a construct as opposed to an object-orientedraising or firing, and may simply be a process occurring within, or asimulation occurring within, a program such as RT client 144.

For example, after a potential change is detected (e.g. after a changetrigger) the graphical image of the GUI may be examined to determine ifthere has been a change over time in the GUI as displayed which updatesa control or control graphic item (or a monitored control graphic item).A change trigger may cause such an examination or search for changes:for example a timeout or the end of a repeating time period may be achange trigger. A timeout or the end of a time period may be considereda change trigger in and of itself, or in combination with a comparisonof a current screen image to a past or cached screen image which resultsin the detection of a difference between the two images. A changetrigger may include the detection of keyboard input (e.g. on the clientdevice), the detection of mouse input or another indication that the GUI(e.g. on the client device) has changed may trigger a search forchanges.

As a result of this search, actions taken by or on controls or GUIelements may be detected (e.g. a button being clicked, text being addedto a text box by a user or an automatic process). Based on theseactions, simulated instantiated software objects (typically operated bysoftware or application such as RT client 144) may provide output, orthe properties, state or data associated with software objects orsimulated instantiated objects may change (the change typicallycorresponding to the change in the control graphic item): for example anevent may be raised, or fire (typically a simulated event), or the dataor properties reflecting text in a control may change. In someembodiments, software objects are not actually instantiated, and theobjects are an internal representation or fiction that RT client 144uses for itself to access visual on-screen objects; RT client then maycommunicate with a process on a third-party server the text in the GUItextbox displayed by application 142 using such internal (to RT client144) representation. In other embodiments, data for software objects maybe accessible by methods or other devices: for example, a “GetText”method associated with a text box may be called by for example a processon a third-party server, and the GetText method (e.g., operated by orpart of RT client 144) may return the text in the GUI textbox displayedby application 142. Simulating the functionality of the graphicalcontrol may be based on or may be performed after detecting the type ofthe graphical control.

One or more different methods may be used as change triggers or signalsto cause an analysis of an application display for relevant changes tothe GUI or graphical image. It is noted that a change trigger may be inresponse to a change in the application image which does not affect acontrol or GUI or other event of interest, and thus a change trigger maynot result in any event being raised or object properties being changed.Changes may be monitored at the level of the operating system of thecomputer outputting the GUI display. For example, window events hooksand subclass window procedures for handle redraw messages for thedisplay of application 142 may be accessed by RT client 144, and when itis detected, via these messages, that the display of application 142 byuser device 140 (e.g., a graphics card on device 140) has changed, thedisplay may be analyzed to determine which if any controls have changed.This may be performed by a detection of drawing commands altering thevirtual window displayed on device 140 showing the GUI for software 142.Similarly, GDI drawing functions may be intercepted via RT client 144:for example a Redraw event may be raised when the target image or GUI isaltered. GDI intercept may require a native dynamic link library (DLL)injection into the target (e.g. client, such as Citrix) application'smemory space; registering system level breakpoint handlers for all APImethods relevant for windows redraw; using an inter-processcommunication mechanism intended to deliver redraw event information; orthe use of a created object which raises events with the relevantparameters on the occurrence of a redraw event. Input devices 165 ofuser device 140 may be monitored. For example, application 142 maymonitor keyboard and mouse input via hooks, and upon indication that akeystroke or mouse movement or click has been made, the GUI may beanalyzed for changes. A timer or timeout function may cause the GUI tobe analyzed for changes every repeating time period, or if a certainamount of time has passed since the last check for changes.

One or more different methods may be used to determine changes orupdates in a GUI display. Some of those methods are discussed above withrespect to FIG. 9. Further methods are discussed below.

Monitoring of the GUI state and changes in the state may be based onreal time difference calculation of two state application images, forexample a cached, saved or prior image, and a current image. Differencesmay include for example data selection (e.g., background change, borderaround text, etc.) and content data changes or updates (e.g. change intext within a text box). To identify these changes a number of methodsmay be used. In one embodiment, three processes are used in sequence,each providing output to the next: an image common difference regionsidentification process, providing output to a filter background changeregion, in turn providing output to a filter data change regionsprocess. In one embodiment, two different types of image changes may bedetected and used: changes in data selection (e.g., check/uncheck abutton), which can be detected by background or other changes; andchanges in content or data (e.g., changing the content of text).

Applying an image common difference regions identification process(e.g., with mode dilate), GUI images (e.g. one before a change trigger,or an actual or suspected change in the screen, and one current) may becompared using black-white image of absolute differences calculations,such as using the OpenCV Function AbsDiff. A morphology algorithm may beused that smooths pixels shown as changed in the resulting differenceimage (“Difference Image”). The process may use an erode algorithm toobtain region rectangles of changes: rectangular subsets from theoriginal image in which there has been a change. The Difference Imagemay be processed by the OpenCV Function MorphologyEx with the parameterERODE to use an erode algorithm, then on the resulting image, contoursmay be found using for example the OpenCV FindContours Function. Foreach contour a bounding rectangle may be calculated (e.g. OpenCVBoundingRect Function). In one embodiment, rectangles that are below asize threshold, e.g., with width less than two pixels, are removed fromthe rectangle set thus obtained. A number of rectangles, indicatingchanges, or surrounding changed portions, may be thus produced.

The output of image differences processing (a set of rectangles) may befed to background change regions identification, which may identify foreach rectangle of the set of rectangles the type of change: data orcontent changing (which may cause the process to return “false”), asopposed to background changing (which may cause the process to return“true”). This process may recognize changes in black spots: if there areno black spot changes it may be determined that background has notchanged. A process may eliminate the rectangles (input from commondifference regions) without background changes, leaving only rectanglesindicating data changes, in a background change regions identificationprocess. For the remaining rectangles which indicate data changes, theprevious cached image and the current image may be compared with respectto these rectangles: a sub-image pair of rectangles is compared usingfor example a common difference regions method. If any rectangles areoutput, a filter is used to subtract from the sub-images backgroundchanges, to return the data changes in the rectangles.

By creating software objects or control objects, for example havingassociated therewith methods, properties, and events, an embodiment maysimulate an API to a client application. In one embodiment the softwareobjects or control objects are not instantiated and executed, but rathersuch objects are used as a representation within RT client 144, forexample an object construct. Thus, various software objects, or controlobjects, may be defined at design time and used internally by RT client144 at runtime. In other embodiments such objects may be actuallyinstantiated. These software objects may be similar to or mimic actualWindows controls APIs. A non-limiting set of GUI visual objects andcorresponding software objects or control objects is described below;other objects may be defined and used.

A generic control may define events, methods and properties which allcontrols may inherit, according to object-oriented methods. For example,a generic control may include the properties or methods of controlcoordinates relative to the application image (e.g. the location of therectangle of the control image); the actual control image (the image ofthe control itself cropped from the application image); certain designtime properties such as the control name, control state, and what kindof control it is, its dimensions; image preprocessing which may preparethe image for OCR, such as digital noise reduction; mouse and keyboardhook settings, which may be hooks to for example the Windows SDK; and ahistogram equalization method that may perform preprocessing to improvethe contrast in an image.

For example, a Button or Link object may be both a visual objectdisplayed on a GUI and a corresponding button or link software object(or object construct used within RT client 144). Each may be similar tothe standard button or link object used in many GUI environments, withsome additional functionality per embodiments of the invention. As withother controls or GUI elements discussed herein, each button or linkobject appearing on a GUI screen may have a separate internal or“instantiated” object within RT client 144 corresponding to the buttonor link, or a software representation or object construct. In someembodiments, each screen object may be represented, in RT client 144, asan object construct including data and description related to theon-screen object; in some embodiments such an object construct can bethought of as similar to an instantiated object-oriented object. Ingeneral, each GUI graphic item of interest may have a separate object(e.g., object construct) corresponding to it executed, instantiated or“imaginary” within RT client 144. In addition, each object or objectconstruct may inherit standard properties, methods and events from ageneric control.

Such a button or link software object may include methods (in the objectoriented programming sense) of for example Click, which may calculatethe control center (the center of the rectangular image of the control)and simulate a left mouse click on the control center in order to setthe focus on the associated button or link with for example theSendInput command; and Get Caption to perform OCR recognition of thecaption or label of the control from the control image and return therecognized text. Such a button or link software object may includeevents (in the object oriented programming sense) of for exampleClicked, fired on a mouse button left up within the control rectangle(the area of mouse click input). Fired events may be processes thatcommunicate occurrences to, e.g. third party software, e.g., RT server188. Properties (in the object oriented programming sense) of suchobjects may include for example, enabled\disabled. Properties may bevalues or data available, via the relevant object, to third partysoftware, e.g., RT server

A process to determine if the object is enabled or disabled and thusdetermine the property enabled\disabled may include for example:creating a ‘gray’ image from the control image (the displayed image ofthe control) with a defined color (e.g. mostly gray, or a uniform grayimage) for each pixel and having the same size as the control image.This gray image may be compared to a gray reference image defined by theuser at design time as having the button or control enabled or disabled.At design time, a ‘dtnorm’ value may be calculated which is the relativedifference norm for the control's design time control image or referenceimage and the defined color image (e.g. the uniform gray image), forexample on a pixel by pixel basis. If the design time reference image isdefined as enabled, and the relative difference norm for the run timecontrol image and the uniform ‘gray’ image (e.g. ‘rtnorm’ value) isgreater than dtnorm, the button or control at run time may be defined asenabled; otherwise the button or control is defined as disabled. If thedesign time reference image is defined as disabled, and the relativedifference norm for the run time control image and the uniform ‘gray’image (e.g. ‘rtnorm’ value) is greater than dtnorm, the button orcontrol at run time may be defined as disabled; otherwise the button orcontrol is defined as enabled.

The relative difference norm may be for example a number distilled froma calculation across all pixels in compared images. Calculating therelative difference norm may be performed using for example:

$\sqrt{\sum\limits_{I}\;\left( {{{src}\; 1(I)} - {{src}\; 2(I)}} \right)^{2}}\text{/}\sqrt{\sum\limits_{I}\;{{src}\; 1(I)^{2}}}$

Where a square root is calculated for a sum of each corresponding pixelfor src1 (e.g. the image of the control obtained at design time or runtime) less the corresponding src2 (a uniform gray image) pixel squared,divided by the square root of the sum of each pixel, squared in src1.

For example, a TextBox object may be both a visual object displayed on aGUI and a corresponding TextBox software object. Each may be similar tothe standard TextBox object used in many GUI environments, with someadditional functionality per embodiments of the invention. Such aTextBox software object may include methods of for example Get Text,which may perform OCR recognition of the corresponding text boxdisplayed in the GUI control image (the control image may be thegraphical image displayed on the screen showing the control, a subset ofthe overall image) and return the text; and Set Text, which may allowthe caller (e.g., a process on third party server 180) to insert text inthe corresponding text box on the GUI. For example, processes such as RTserver 188 on a third-party system or RT client 144 may take somecontrol of monitored application 142 by causing text to appear in thedisplay of monitored application 142. RT server 188 may do this usingcalls to software such as RT client 144. Manipulating controls by thethird-party software, e.g. executed on third party server 180, may allowfor example a prompt, instructions, or advice to be given (e.g.,automatically) to an agent operating monitored application 142, mayallow autofill of fields, or other control of monitored application 142.

TextBox software object methods may include for example GetText (obtainthe text in the control using for example OCR); and SetText andAppendText, which may use windows SendInput to set or add text(respectively) to the displayed control textbox. SetText and AppendTextmay use a calculation of the relevant control center and may cause anexecution of a LeftMouse click (set focus) call in order to bring focusto the relevant control, and such methods may use the Windows SendInputcommand (or a similar command, which simulates keyboard typing), andexecution of {CTRL}A, and then {DEL} (select all, and then delete).SendInput or AppendText methods may thus insert or add (respectively)text into the corresponding displayed text box control.

A TextBox software object (e.g. corresponding to a graphical controlallowing input of text) may include events including Text Changed, whichmay fire or be raised when for example RT client 144 detects that textin the text box has changed or updated, notifying for example a processsuch as RT server 188 on third party server 180 that text has changed.This may be fired for example when the previous control state imagestored or cached is found to differ from the current control state imagein a relevant way. Such a change may be detected, for example, bycalculating text changes (when presented with a set of changed areasrectangles) based on relative difference of current control image(typically the graphical image displayed on the screen showing only ormostly the control) with the image stored or cached, using the FilterData Change Regions Method described above. If rectangles not are“empty” (empty indicating no change) then an event is fired.

A CheckBox (e.g., that permits the user to make a binary choice) orRadioButton (e.g., allowing input of a choice of only one of a set ofmutually exclusive options) object may be a visual object displayed on aGUI and a corresponding software object. Each may be similar to thestandard CheckBox or RadioButton objects used in many GUI environments,with some changes according to embodiments of the present invention.Such a software object may include a method of, for example, GetText,which may perform OCR recognition of the corresponding text displayed inthe GUI for the object control image and return the text. Such asoftware object may include a property of, for example,checked/unchecked. Such a software object may include an event of, forexample, a checked/unchecked state change or update, typically fired ona state change or upon the mouse-left-up causing the corresponding statechange, e.g. from checked to unchecked or vice versa.

An algorithm to detect a checked state change (for example for use witha CheckBox or RadioButton) may obtain a bi-level (e.g., binary) image,and apply a threshold function to compare each pixel intensity value athreshold value. Edges detection may use a Laplace operator (e.g., theLaplacian may use the gradient of images). Contours may be identified,and an approximate polygon and rectangle for each contour may becalculated. Then two regions in the CheckBox or RadioButton image may beidentified: the check mark and the text. A check mark area imageidentified at design time (known to have a certain state, e.g., checkedor unchecked) may be matched or compared with such an image updated atruntime.

A Combination Box or ComboBox object (e.g. a box containing acombination of controls, such as sliders, text boxes, and drop-downlists) may also be a visual object and a corresponding software object.Such a software object may include methods of, for example, GetText(returning text produced from OCR recognition); and Select Index (whichmay perform the functions of determining the location of the controlcenter, executing a Left Mouse click (to set the focus, in order tobring focus to the relevant control) with the SendInput command tosimulate a press of the down arrow in the on-screen control n times,where n is the index of selection. Methods may include SelectItem where,for an editable combo box, keyboard input may be sent for control-A(select all) then delete, and then text data and Key “ENTER” may besent, and for a non-editable combo box only text data may be sent bykeyboard simulation, followed by the enter key simulation. Such asoftware object may include events including Selection Changed,returning output that the selection of the object has changed orupdated. This may include finding the location of the combo box itemimage (e.g. using a Detect Item Location algorithm); then calculation oftext changes (where the text change calculation is first presented witha set of changed areas rectangles) based on a relative difference of thecurrent state control image with the image cached, e.g. the prior imagesaved, using for example a Filter Data Change Regions Method describedabove. If the rectangles are not empty (empty rectangles indicating nochange) then an event is fired.

A detect item location algorithm used with a ComboBox object mayinclude, for example, first, obtaining a bi-level (e.g. binary) image. Athreshold function may compare each pixel intensity value with athreshold value. Edges detection may be performed, for example using aLaplace operator (the Laplacian uses the gradient of images). Contouridentification may be performed, and calculation of an approximatepolygon and rectangle for each contour may be performed. Two regions inthe ComboBox image may be identified, for example Button and Text. Theselected item image may be cropped from the text area image of thecontrol.

A ListBox object (a visual object and a corresponding software object)may include methods of, for example, SelectIndex, SelectItem andGetSelectedText. Select Index may perform the functions of calculationof or finding the control center (the center, on the screen, of thecontrol), execution Left Mouse click (to set focus in order to bringfocus to the relevant control), using a SendInput command to simulate apress of the “home” key and the down arrow (the depiction of the downarrow in the on-screen control) control button n times, to move theselection downward, where n is the index of selection. The SelectItemmethod may cause the simulation of an item selected in a list (e.g.,third party server 180 may via RT server 188 cause a selection to besimulated, via RT client 144 in application 142). This may occur forexample by extracting text with their screen or relative coordinates) byfor example OCR, and retrieving the item selected by analyzing the dataof the words extracted. If the item is found (e.g. if the text on thescreen matches the expected text for the item to be selected) then theappropriate mouse click may be simulated (as discussed elsewhere) toselect the item in the relevant application. The GetSelectedText methodmay return the text selected, possibly using a Selected Item Detectionalgorithm (discussed below) or another suitable algorithm. The ItemSelected event may fire if it is determined that the item selected isdifferent than the previously selected item; this may be detected bydetecting changes in the text background color.

The Selected Item Detection algorithm discussed above may include forexample creating a Histogram Equalization of the control image;performing edges detection with for example a Canny edge detector; usinga Hough transform or other suitable technique for finding horizontallines; identifying two regions in the ListBox Image, for example Listbox and Scroll bars areas; processing a selected rectangle withBackground Change Regions Identification Method (discussed elsewhereherein), and finding lines. Then, cropping may be performed on thesub-image of the ListBox Item selected from the list box control image.

In order to control or receive input from any region of the GUI display,beyond those specified by controls, for an application (e.g.,application 142), in one embodiment, any region of the GUI display orapplication image may be considered to be part of a Generic Control,which may have a corresponding software object. A Generic Controlsoftware object may include and fire control actions or events such asClick, Double Click, Get Text and other appropriate events to causeevents on the screen of a monitored application. Such an object mayinclude a set of identification attributes common for all other controlson the screen, as in some embodiments all controls inherit genericproperties, methods and events from a common control. Such a softwareobject may include methods of, for example, Get Text, which returns textobtained by for example OCR recognition from the control image;SendInput, which may create or synthesize keystrokes, mouse motions andclicks; and Scroll, which may for example call a Raise Mouse clickfunction on scrollbars or emulate ‘Page Up/Down’, ‘Up/Down’ keystroke.In order to determine the location of scroll bars and scroll barbuttons, a Scroll Bars Location method or algorithm may be used. Ascroll bar software object may include events of, for example, MouseClicked, which may fire on the occurrence of a Mouse Left Button up in acontrol rectangle.

A “Scroll Bars Location” algorithm, to determine the location of ascroll bar, may be used in conjunction with or as part of a GenericControl. Such an algorithm may include, for example performing ahistogram equalization of the control or GUI image; performing edgedetection using for example a Canny edge detector; using a Houghtransform or other technique for finding horizontal and vertical lines;and identifying scroll bar buttons areas in the GUI or control image,depending on the direction of the scrolling.

At runtime any desktop application may be identified by its framewindow. A visual display on an agent's monitor (e.g. on user device 140)may include a number of windows unrelated to the monitored or clientapplication (e.g., application 142); further, the monitored or clientapplication may appear within a “hosting” window (e.g. a Citrix window)on the agent's monitor. While a window may be found using its “handle”or other known processes using windows, this does not apply for thirdparty software which has no access to the “hosting” window (e.g. aCitrix window) messages. If the monitored application is running inremote desktop environment, its frame windows cannot be recognized byregular window recognition methods. Thus, a monitoring process such asRT client 144 may be able to find the hosting window (surrounding themonitored application image) easily using e.g. its window handle, thetypical API may not be available for the monitored application. Themonitored application's “frame window” may need to be identified withinthe hosting window using its visual, graphical, image. For this purpose,an object called Frame Control (which can be considered as VirtualWindow) may be instantiated (as with other objects, “instantiated” maybe the equivalent of a virtual construct used in a monitoring process).The frame control object may contain all controls and anchors definedfor the application and may be found or identified by anchors andcontrols contained in it. Anchors may be visual aspects of the clientprogram that do not change, such as the title of the program.

A Frame Window object may represent the monitored application andinclude methods such as Get Caption, which may use OCR to recognize thecaption (e.g. the text area at the top of the application frame, or thetopmost text) of the image. As part of Get Caption, caption and buttonslocations may be searched using for example a Hough Transform techniquefor finding horizontal and vertical lines. A Minimize method maysimulate a left mouse click with the SendInput command on the window'sminimize button to minimize the client window, and a Maximize commandmay simulate the left mouse click with SendInput command on maximizebutton to maximize the window. A Close method may simulate a left mouseclick with the SendInput command on close button. A Bring To Frontmethod may bring the application window to the front if other windowsare over the frame window. A Bring To Front method may determine if theapplication is an icon, and if so the method may set the focus on theicon, then execute or simulate an Alt-Tab keystroke which may show alist of application icons. A screenshot image may be obtained, and theicon location may be found within the image. A mouseclick may then besimulated to maximize the client application. If the Bring To Frontmethod determines the application is not an icon, the design time imagemay be compared to the current frame image, and if the features match, amouse click may be simulated to bring the client application to thefront.

A frame window object may include properties such as icon, which is theApplication image icon, which may be the icon (small visualrepresentation) of the client application when it is minimized, and itsicon image appears on the screen.

A frame window object may include events such as mouse clicked which mayfire event on Mouse Left Button up in the control rectangle.

FIG. 12 is a flow diagram of a method for enabling interoperability oran API with a run-time application according to an embodiment of theinvention. Elements and operations of the embodiment of FIGS. 12 and 9may be used together if appropriate, and may be used with components ofFIG. 1, or another system. In operation 1300, software linking to orcommunicating with a third-party application (e.g., a third partyapplication operated on third party server 180) may be installed on auser device (e.g., user device 140). For example, RT client 144, e.g.embedded software, may be installed and executed on a user device tomonitor and communicate with target or monitored application 142, and toprovide an API from monitored application 142 to RT server 188. Targetor monitored application 142 may be considered a computer executedapplication operated on a computer system, and RT client 144 may also beexecuted on the same computer system to examine the monitoredapplication image or GUI produced by or displayed by the target ormonitored application. In one embodiment, the computer executing aremote, third party program (e.g., RT server software modules 188) andthe computer executing the agent application 142 may be remote from eachother, and the two applications may communicate via RT client 144.

RT client 144 may take as input a scene. In some embodiments, this maybe done by compiling a scene (containing for example informationregarding control images and controls) into a DLL, which may beinstalled on the user device 140 on which an RT client 144 executes, andthe RT client 144 may access the DLL to locate and provide the interfacewith the objects defined by the scene. For example, a native DLL may beinjected into the target application memory space; system levelbreakpoint handlers for all API methods relevant for windows redraw maybe registered on a user device hosting the target application; aninter-process communication mechanism may be created intended to deliverredraw event information to for example software such as RT client 144or to the third party software intended to receive the redrawinformation; and/or an object may be created at design time representingthe relevant API which raises events with the relevant parameters for RTclient 144 or to the third party software intended to receive the redrawinformation. The embedded software may be defined by or controlled bydesign time data, such as a “scene” as discussed elsewhere herein, whichin some embodiments may define the actions of RT client 144 by beingcompiled into a DLL which is used by RT client 144.

In operation 1310, an initial application image, graphical image, or GUIdisplay may be captured, an application image scene may be recognized,e.g. among the various windows on a received image display, the targetclient application image may be found, and an application image,graphical image, or GUI display may be captured and cached or stored forcomparison with an image captured in the future. FIG. 9 depicts anexample embodiment for such application image scene recognition. Suchrecognition may be performed both at start up and also if it isdetermined that an anchor as changed.

In operation 1320 on the occurrence of a change trigger or an indicationthat a screen change significant or relevant to an API may haveoccurred, or in the event that it is determined that an anchor wasupdated, an additional application image, graphical image, or the GUIdisplay for the monitored application is captured and stored or cached.The image captured in operation 1320 may be considered the currentimage, and the image captured previously (e.g. in a previous operation1320 or initially in operation 1310) may be considered the prior image;over time or iterations the current image may become the prior image. Achange trigger may be for example some event or message that indicates ascreen or image change has taken place that might be relevant to the APIor controls on the screen. Such a change trigger can be for example oneor more of the occurrence of the end of a time period or the expirationof a timer, the detection of keyboard, mouse, touchscreen or other inputwhich may cause the change in a screen image, the capture orinterception of an appropriate operating system or graphics systemmessage, or the indication that the GUI or client display has changed.Certain embodiments may use only one of these types of change triggers,and other embodiments may combine more than one of these changetriggers, and capture an image on the occurrence of any such changetrigger. Other change triggers may be used.

For example, a change trigger may be the receipt by RT client 144 of amessage related to a window events hook and subclass window procedurefor handle redraw messages, possibly obtained from the graphics card onthe client computer operating RT client 144. A change trigger may be thereceipt by RT client 144 of a message intercepted from GDI drawingfunctions. A change trigger may be the receipt by RT client 144 of amessage intercepted from monitoring keystrokes and mouseclicks, or otherinputs (e.g., a press on a touch screen) on the target client computer.In other embodiments other processes, such as processes on a third-partyserver, may receive the change triggers.

An application image, graphical image, or the GUI display provided by amonitored application, such as application 142, may be captured orobtained, typically be monitoring software such as RT client 144. Forexample, a screenshot may be obtained. The application image, graphicalimage, GUI display, screenshot or screen image may be saved or cachedfor comparison with a future obtained screenshot or screen image.Typically only the image of the terminal or remote client hosting window(e.g., Citrix) is captured, as opposed to the entire image displayed onthe client computer.

In operation 1330 the graphical image (e.g., the current image) of themonitored GUI may be analyzed or examined (e.g. by comparison with theprior or immediately-in-time prior captured, cached or stored image) todetermine if there has been a change or update over time (e.g. bycomparing the image to a previously cached image). If there is no updatethe process may return to operation 1320 to wait for another trigger orchange event. If there is an update the process may proceed to operation1340.

In operation 1340, if there was a change, it may be determined if ananchor was updated or changed. If it is determined that an anchor wasupdated, this may indicate that the change in image was for example theclient application being moved, minimized, maximized, etc., and theprocess may move to operation 1320 to capture an image. If it isdetermined that an anchor was not changed, the process may move tooperation 1350.

In operation 1350 the graphical image (e.g., the current image) of themonitored GUI may be analyzed or examined (e.g. by comparison with theprior or immediately- in-time prior captured image) to determine ifthere has been a change or update over time to controls (e.g. bycomparing the image to a previously cached image), or a change over timethat affects the API or the relevant monitored controls or other aspectsof the client software, or since the last time the image was examined,in the GUI as displayed. In particular, examination may determine ifthere was a change which updates a control graphic item, or a monitoredcontrol graphic item, e.g. a change in the visual depiction of agraphical control. This may be performed, for example, by embeddedsoftware (e.g. RT client 144) installed on a client device. In someembodiments, when determining if a change occurred that affects acontrol, for each control, a template image determined at design timemay be matched to the control images during runtime to locate eachcontrol.

If there has been no change over time, or if there is no change tocontrol graphic items, or to monitored control graphic items, theprocess may wait for another trigger, e.g. iterate (operation 1320).

If there has been a change over time, or a change or update to controlgraphic items, or to monitored control graphic items, the process may inoperation 1360 take an action with respect to the control graphicitem(s) which have been determined to have changed or updated since thelast image was obtained. An action may include, for example, updating orchanging properties (e.g., object oriented properties accessible byquerying the instantiated object, or simply data associated with aninternal representation (“object construct”) of a screen object) of asoftware object or instantiated object, or an internal representation,associated with or representing the changed control graphic items, orraising or executing an event (e.g., an object-oriented event, afunction which sends a message on certain conditions where the eventsource—e.g., the control software object—sends a message via an event toa listener or event handlers—e.g. the third-party software) or otherwisesending a signal or message corresponding to the software object.Raising an event may include notifying a process (e.g., a portion of RTclient 144, or a process external to RT client 144, such as RT serversoftware modules 188, of the change. For example, an instantiated objector an internal construct (e.g. object construct) associated with thecontrol that has been changed or updated may fire an event or otherwisesend a message. The event may communicate information, such as theoccurrence of a mouseclick or screen touch, a change in state of awindow, or the change to text, to the third party application. Forexample, RT client 144 may communicate state changes of application 142to a third party application executed on third party server 180, vianetwork 105. Ways of communicating the alteration of the properties of avisual object may be used other than object-oriented event or propertiestechniques.

The process may continue by waiting, e.g. iterate (operation 1320), forthe occurrence of a change trigger or an indication that a screen mayhave changed in a way relevant to a GUI.

At any point in the process of FIG. 12, in operation 1370, a third partyapplication may affect the state or provide input to the monitoredapplication. For example, a third party application may send a messageto, call a procedure or execute a process, in order to gatherinformation from or change the state of the monitored application. Thismay be done by an object oriented method call, but may be done by othersoftware methods. For example, an application executing on third partyserver 180 may cause a message to a human agent to appear on the GUIoutput of application 142 via making a method call to a control (e.g. toan instantiated object or an internal construct associated with thecontrol) of RT client 144. Other or different operations may be used.

Reference is now made to FIG. 13 which is a flowchart of a method forenabling graphic-based interoperability with an application executed ona computer (e.g., system server 110) remote to another computer (e.g.,user device 140) including a local processor (e.g., user processor 145),and displayed on a local display (e.g., output devices 170), accordingto embodiments of the invention. While in some embodiments theoperations of FIG. 13 are carried out using systems as shown in FIG. 1,in other embodiments other systems and equipment can be used. Accordingto some embodiments, the operations of FIG. 13 are carried out using thelocal processor.

According to some embodiments, operations 1380-1386 may be performed asa preparation step during a design-time mode. In operation 1380, thelocal processor may receive a design-time application image, e.g.,similarly to step 205. In operation 1382, the local processor mayreceive a control object located within the design-time applicationimage. e.g., similarly to step 245. For example, the local processor mayreceive the control selection as input from a user. In some embodiments,the control may be associated with a control type. For example, thecontrol type for the control may be identified, determined, recommended,and/or suggested automatically by the local processor and/or theprocessor may receive a control type selection as input from a user. Inoperation 1384, the local processor may identify or determine an anchorfor the control object. According to some embodiments, identifying theanchor for the control object may include selecting an image of text oricon that is located inside a bounding shape object that defines thecontrol or is outside of the bounding shape object that defines thecontrol and nearest to (e.g. on the display or screen) the boundingshape object that defines the control. In operation 1386, estimatedlocation of the anchor in the design-time application image may bestored. According to embodiments of the invention, geometricrelationships between a control and the anchor may be calculated ormeasured and stored as well. For example, the location of the anchorwith relation to the location of the control may be measured oridentified and stored. Thus, when the anchor is later identified duringrun-time, the location of the control relatively to the anchor may beretrieved and the control may be identified in the run-time applicationimage, and possibly used to manipulate the control during run-time.

For example, identifying by the processor an anchor for a control objectthat is one of a button or a link may include selecting a content imagelocated inside a bounding shape object that defines the control, e.g., abounding rectangle or another shape. A content image may includegraphics or content located inside a bounding shape object, e.g., abounding shape object that defines the control or a bounding shapeobject that is nearest to the control on the application GUI, e.g.,icon, or text. It may be assumed that the graphical content of theanchor would not change from design-time mode to run-time mode.According to some embodiments, an image of the graphical content of theanchor is stored as an anchor. Thus, when the anchor includes text, thenaccording to one embodiment, an image of the text may be stored as theanchor. For example, FIG. 14 depicts a part of a design-time applicationimage 1440 with an OK button control 1410, a cancel button control 1420,and an anchor 1422 identified for the cancel button control 1420,according to embodiments of the invention. In this example, anchor 1422is a bounding rectangle. Thus, an image including the word “Cancel” asit appears on the GUI of the design-time application image 1400 may bestored and later used as anchor 1422. FIG. 15 depicts a part of adesign-time application image 1500 with an example of a link control1510, and an anchor 1512 identified automatically for link control 1510,according to embodiments of the invention. For example, when linkcontrol 1510 is pressed during run-time mode, the Gmail application mayopen. In this example, anchor 1512 identified automatically for linkcontrol 1510 is a bounding rectangle of the graphics of the link as itappears on design-time application image 1500, e.g., the graphics of thephrase “Hyperlink1—Click Me”.

According to some embodiments, identifying by the processor an anchorfor a control object that is one of check box and a radio button mayinclude selecting content image located nearest (e.g. when measured onthe display, for example in units of pixels or other distance measures)to the control. FIG. 16 depicts a part of a design-time applicationimage 1600 with an example of a check box or radio button control 1610and an anchor 1612 automatically identified for radio button control1610, according to embodiments of the invention. In this example, theanchor 1612 automatically identified for radio button control 1610includes a bounding rectangle of the text closest to radio buttoncontrol 1610. Thus, an image including the word “Option1” as it appearson design-time application image 1600 may be stored and later used asanchor 1612.

According to some embodiments, identifying by the processor an anchorfor a control object that is a scrollbar object may include selectingimage of a scrollbar button. FIG. 17 depicts a part of a design-timeapplication image 1700 with an example of a scrollbar control 1710 andan anchor 1712 automatically identified for scrollbar control 1710,according to embodiments of the invention. In this example, the anchor1712 automatically identified for scrollbar control 1710 includes animage of a scrollbar button. Thus, an image including the scrollbarbutton as it appears on design-time application image 1700 may be stored(e.g., in database 135) and later used as anchor 1712.

According to some embodiments, scrollbar controls may be identified orfound by the processor using, for example operations such as:

-   Obtaining an application image (e.g., a design-time application    image).-   Creating a pattern dictionary of a variety of images of scrollbars,    including for example horizontal or left-right scrollbars and    vertical or top-bottom scrollbars.-   Binarizing the images of the scrollbars for example by applying    AdaptiveThreshold function of the OpenCV library.-   Creating a Sels dictionary for possible scrollbars by generating a    binary structuring element (Sel) for each image of a scrollbar, for    example, using functions such as PixGenerateSelRandom or    PixGenerateSelWithRuns from Leptonica SDK (Leptonica is an open    source library containing a set of functions for processing and    analysis images).-   Binarizing the run-time application image for example using OpenCV    function AdaptiveThreshold and converting the result to Leptonica    binary format.-   Finding locations in the run-time application image that match the    patterns from the Sels dictionary for example, using Leptonica based    function PixFindMatchedPatterns.-   Finding the nearest left and right arrows or forelocks pairs for    horizontal scrollbars and top and bottom arrows or forelocks pairs    for vertical scrollbars.-   Finding or determining a bounding shape for each scrollbar, e.g.,    scrollbars rectangles.

FIG. 18 depicts a part of a design-time application image 1800 with anexample of scrollbars and scrollbar rectangles, according to embodimentsof the invention. For example, scrollbar rectangle 1812 is identifiedfor horizontal scrollbar 1810, and scrollbar rectangle 1822 isidentified for vertical scrollbar 1820.

According to some embodiments, identifying an anchor for a controlobject that is a table may include selecting an image of content of aheader cell. FIG. 19 depicts a part of a design-time application image1900 with an example of a table control 1810 and anchor 1912automatically identified for table control 1910, according toembodiments of the invention. In this example, the anchor 1912automatically identified for table control 1910 includes an image of acontent of a header cell, e.g., the top left header for languages thatare written from left to right. Thus, an image including the content ofa header cell as it appears on design-time application image 1900, inthis example, an image of the word “Name”, may be stored and later usedas anchor 1912.

According to embodiments of the invention, identifying an anchor for acontrol object may include selecting an image of text or an icon that islocated inside a bounding shape object that defines the control or isnearest to the bounding shape object that defines control. For example,selecting an image of text or icon that is located nearest to thecontrol may be performed for controls or types of controls for whichembodiments described with relation to FIGS. 14-19 have failed to detector identify an anchor. According to some embodiments, selecting, by theprocessor, an image of text or icon that is located nearest to thecontrol may include performing example operations such as:

-   Binarizing the design-time application image for example by applying    AdaptiveThreshold function of the OpenCV library.-   Detecting bounding rectangles of words and icons in the binarized    design-time application image, for example, by applying Leptonic    GetWordsInTextlines function (e.g., a function that returns    rectangles of words in an image).-   Excluding rectangles of other controls and shapes of the same level    in the shape tree from the list of rectangles (e.g., shapes of other    controls).-   Finding the nearest bounding rectangles, typically by scanning the    design time image from the left to the right and from the top to the    bottom. Typically, a control located nearest to the top-left corner    of the control may be selected.-   For right to left languages, e.g., Hebrew and Arabic, the search    sequence may be from the right to the left and from the top to the    bottom. Typically, a control located nearest to the top-right corner    of the control may be selected.

FIG. 20 depicts a part of a design-time application image 2000 with anexample of detected bounding rectangles of words and icons 2020,according to embodiments of the invention. In this example, the anchor2012 automatically identified for window 2010 includes an image of acontent of bounding rectangle 2012, e.g., the bounding rectangle locatedclosest to the top left corner of window 2010. Thus, an image includingthe content of bounding rectangle 2012 as it appears on design-timeapplication image 2000, in this example, an image of the word “Tree” maybe stored and later used as anchor 2012.

According to some embodiments, operations 1388-1392 may be performedduring normal and routine operation of the application, e.g., atrun-time mode. In operation 1388 a run-time application image displayingon the local display may be captured. According to some embodiments,operation 1388 may be performed periodically, e.g., each 0.5, 1 or 1.5seconds and/or when a change is sensed in the GUI. In operation 1390, ananchor, e.g., the anchor identified in operation 1384 and stored inoperation 1386, may be identified, found or detected within the run-timeapplication image. For example, the anchor may be identified in therun-time application image using the matchTemplate function from theOpenCV library. The matchTemplate function searches for a smaller image(up to same size image) within a larger image and returns the locationof the smaller image in the larger image if found. The matchTemplatefunction may also return a match or similarity score, indicative of thesimilarity between the searched and found anchor. In operation 1392, thecontrol object may be identified within the run-time application imagebased on the anchor. For example, the geometric relationships between acontrol and the anchor (stored in operation 1386) may be used toidentify the control once the location of the anchor is known. Aplurality of anchors and respective controls may be found.

Reference is now made to FIG. 21 which is a flowchart of a variant ofthe method for enabling graphic-based interoperability with anapplication executed on a remote computer (e.g., system server 110)remote to a local computer (e.g., user device 140) including a localprocessor (e.g., user processor 145), and displayed on a local display(e.g., output devices 170), according to embodiments of the invention.While in some embodiments the operations of FIG. 21 are carried outusing systems as shown in FIG. 1, in other embodiments other systems andequipment can be used.

According to some embodiments, the operations of FIG. 21 are carried outusing a local processor. According to embodiments of the methodpresented in FIG. 21, operations 1380-1392 are carried out similarly asdescribed elsewhere herein, and operation 2110 is added to theembodiments of the method presented in FIG. 13. In operation 2110 anestimated location of the anchor in the design-time application imagemay be stored. According to one embodiment, coordinates of the anchor inthe design-time application image may be stored. According to someembodiments, the design-time application image may be divided intosegments and the one or more segments containing the anchor may beidentified, found or determined and stored. Thus, identifying the anchorwithin the run-time application image in operation 1390 may includesearching for the anchor starting from the estimated location, e.g.,from the stored coordinates or the stored segments. For example, therun-time image may be divided into segments that are equivalent to thesegments of the design-time image, and the anchor may be searched firstin the stored segments (or in an area that is slightly larger than thestored segments). According to some embodiments, dividing thedesign-time or run-time application image into segments may includedividing the design-time or run-time application image into N*Nrectangular segments, where N is an integer larger than one. FIG. 22depicts a part of a design-time application image 2200 that is dividedinto segments 2210, according to embodiments of the invention.

Reference is now made to FIG. 23 which is a flowchart of a variant ofthe method for enabling graphic-based interoperability with anapplication executed on a remote computer (e.g., system server 110)remote to a local computer (e.g., user device 140) including a localprocessor (e.g., user processor 145), and displayed on a local display(e.g., output devices 170), according to embodiments of the invention.While in some embodiments the operations of FIG. 23 are carried outusing systems as shown in FIG. 1, in other embodiments other systems andequipment can be used.

According to some embodiments, the operations of FIG. 23 are carried outusing a local processor. According to embodiments of the methodpresented in FIG. 23, operations 1380-1392 are carried out similarly asdescribed elsewhere herein, and operations 2310-2330 are added to theembodiments of the method presented in FIG. 13. In operation 2310 theprocessor may determine, identify or infer that the size of the run-timeapplication image is different from the size of the design-timeapplication image, e.g., that the run-time application image isincreased or decreased relatively to the run-time application image.According to some embodiments, the local processor may determine,identify or infer that the size of the run-time application image isdifferent from the size of the design-time application image by matchingor comparing the run-time application image with one or more anchors,and generating a match score indicative of the similarity between therun-time application image and the one or more anchors. For example, amatchTemplate function may be used to compare or match the run-timeapplication image with the one or more anchors and to calculate orprovide the match score. Other methods may be used. According to someembodiments, the local processor may perform operation 2310 for eachrun-time image that is obtained in operation 1388. According to someembodiment, the local processor may perform operation 2310 if the localprocessor fails to identify anchors, e.g., in operation 1390.

For example, a user of the application in run-time may use zoom-in orzoom-out to increase or decrease the size or resolution of anapplication image during run-time according to preferences of the user.Thus, if the application or GUI of the local processor enables zoomingin and zooming out, then the size of the run-time application image maybe different from the size of the design-time application image, andspecifically, the size of the anchors in the run-time application imagemay be different from the size of the same anchors taken from thedesign-time application image, e.g., increased or decreased by a scalingfactor. Thus, trying to detect or identify the anchors within therun-time application image may provide false negative results.Therefore, to identify the anchors that were stored during the designtime, it may be required to match the sizes of the stored anchors to thesize of the same anchors in the run-time application image, e.g. toalter the size of one of the two so that the sizes of the two areequivalent or similar, by some measure. To do that, it may be firstrequired to know the scaling factor, which is not provided from theapplication at run-time.

Therefore, in operation 2320, a resize factor for the run-timeapplication image relatively to the design-time application image may becalculated or found, e.g., by the local processor. In some embodiments,calculating the resize factor may include resizing the run-timeapplication image or the one or more anchors found in the design phase,using an initial resize factor, e.g., an initial guess. According tosome embodiments, resizing the run-time application image or the one ormore anchors may include only zooming out (e.g., decreasing in size)either the run-time application image or the one or more anchors.Zooming in or increasing in size an image may cause visual inference,e.g., blurring of the increased image. Thus, in some embodiments zoomingin one of the images in operation 2320 is avoided. The resized, e.g.,zoomed out or decreased, run-time application image may be matched orcompared to the one or more anchors, or the resized, e.g., zoomed out ordecreased, one or more anchors may be matched or compared to therun-time application image (depending on which image has been resized)and a match score may be calculated (e.g., using the matchTemplatefunction or other method that compares the images and provides the matchscore). The resizing and matching may be repeated with various ordifferent resize factors, e.g., the local processor may perform a searchover different resize factors or over a range of possible resizefactors, until the match score meets a criterion, e.g., until the matchscore is below a threshold. The resize factor that provides the matchscore that meets the criterion may be selected.

According to some embodiments, the local processor may calculate theresize factor by first searching over resize factors that zoom out therun-time application image. The search may be performed by zooming outthe run-time application image using an initial guess of a resizefactor, matching the zoomed-out run-time application image with theoriginal size one or more anchors to calculate a match score andrepeating the resizing and matching with different resize factors. Ifthe match score meets a criterion, the resize factor that provided thematch score that met the criterion is selected. If none of the testedresize factors provides a match score that meets the criterion a failureto find a resize factor by zooming out the run-time application image isdeclared. If a failure to find a resize factor by zooming out therun-time application image is declared, then the local processor maycalculate the resize factor by searching over resize factors that zoomout the one or more anchors. The search may be performed by zooming outthe one or more anchors using an initial guess of a resize factor,matching the zoomed out one or more anchors with the original sizerun-time application image to calculate a match score and repeating theresizing and matching with different resize factors. If a match scoremeets a criterion, the resize factor that provided the match score thatmet the criterion is selected. If none of the tested resize factorsprovides a match score meets the criterion a failure to find a resizefactor is declared.

According to some embodiments the search over a range of resize factorsmay include calculating a resize factor from an interval or range ofpossible resize factors. The search may include performing one or moreiterations, where in each iteration a step size and a range of possibleresize factors may be selected, and one or more candidate resize factorsmay be generated or calculated using the step size and range of possibleresize factors. For example, the searched range may be [0-1], [0-2],etc., and the initial step size may be 0.05, 0.1, 0.2, etc. The one ormore candidate resize factors may be generated as an arithmeticprogression of candidate resize factors that are bounded in the range ofpossible resize factors with a common difference that equals the stepsize. For example, a first candidate resize factor may equal the minimalvalue from the interval or range of possible resize factors, a secondcandidate resize factor may equal the first candidate resize factor plusthe step size, a third candidate resize factor may equal the secondcandidate resize factor plus the step size, etc., until the largestcandidate resize factor that is not larger than the maximal value fromthe interval or range of possible resize factors is reached.

In a first set of iterations, the range of possible resize factors mayinclude candidate resize factors that zoom out the run-time applicationimage. To perform the search, each of the candidate resize factors maybe used to zoom out the run-time application image. Each of thezoomed-out run-time application images may be matched or compared withthe design-time application image and a match score may be calculatedfor each of the candidate resize factors. For each consecutiveiteration, the step size may be reduced by a reduction factor, e.g., bya factor of 5, 10, 20, etc. In some embodiments the range of possibleresize factors may be reduced as well so that the search would continuein the following iteration in a range that is closer to the candidateresize factor that provided the best results in the current iterationthan the range before the reduction. For example, if the range on thefirst iteration is [0-1], the resize factor is 10 and the step size is0.1, than the candidate resize factors of the first iteration would be0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 and 1. If for example,the candidate resize factor that provided the best match score was 0.5,than in a next iteration the range would be 0.45-0.55 and the step sizewould be 0.01. According to embodiments of the invention, reducing thestep size at each consecutive iteration may improve the accuracy of thesearch and reducing the range of possible resize factors at eachconsecutive iteration may dramatically improve the efficiency anddramatically reduce the computational power required for finding theresize factor. Combining reducing the range of possible resize factorswith deceasing the step size at each iteration, embodiments of theinvention may provide an accurate and efficient search.

The iterations may be repeated until a maximal match score of a currentiteration equals (or substantially equals) a maximal match score of aprevious iteration, or until the maximal match score is above athreshold, and the candidate resize factor that provided the maximalmatch score may be selected and used. If the maximal match score in acurrent iteration is lower than the maximal match score in a previousiteration, the algorithm terminates, and it may be determined orconcluded that no resize factor that zooms out the run-time applicationimage was found.

In some implementations, user configuration may allow only zooming in tothe run-time application image by the user during run-time. In thiscase, only the first set of iteration may be performed. In someimplementations, however, the user may either zoom in or zoom out therun-time application image. In this case, a second set of iterations maybe required if no resize factor that zooms out the run-time applicationimage was found in the first set of iterations, which could mean thatthe user had zoomed out the run-time application image.

If no resize factor that zooms out the run-time application image wasfound in the first set of iterations, then a second set of iterationsmay be performed. In the second set of iterations, the range of possibleresize factors may include candidate resize factors that zoom out theone or more anchors. To perform the search, each of the candidate resizefactors may be used to zoom out the one or more anchors. Each of thezoomed-out one or more anchors may be matched or compared with therun-time application image and a match score may be calculated for eachof the candidate resize factors. For each consecutive iteration, thestep size may be reduced by a factor. In some embodiments the range ofpossible resize factors may be reduced as well so that the search wouldcontinue in the following iteration in a range that is close to thecandidate resize factor that provided the best results in the currentiteration. The iterations may be repeated until a maximal match score ofa current iteration equals (or substantially equals) a maximal matchscore of a previous iteration, or until the maximal match score is abovea threshold, and the candidate resize factor that provided the maximalmatch score may be selected and used. If the maximal match score in acurrent iteration is lower than the maximal match score in a previousiteration, the algorithm terminates, and it may be determined orconcluded that no resize factor was found.

In the following example embodiments of the method for finding a resizefactor in either the first or second set of iterations is demonstrated.In this example, the matchTemplate function is used to compare or matchthe run-time application image with the one or more anchors. In thefirst set of iterations the run-time application image is zoomed outusing the candidate resize factors and in the second set of iterationsthe one or more anchors is zoomed out candidate resize factors. Theresult of the matchTemplate function is used as the match score. othermethods may be used to calculate the match score. For example, let x bethe range of possible resize factors where x⊂[0.0,1.0], letmatchTemplate function be denoted as ƒ(x), x⊂[0.0,1.0]. Thus, theproblem may be defined as finding x that corresponds to max [ƒ(x)]. Theinitial step size is selected to be 0.5. In a first iteration, ƒ(x) iscalculated for x=x+step_size in [0.0,1.0]. Further, let n denote theiteration number. In each iteration, a maximal match score, denotedmaxvalue(n) is calculated. If maxvalue(n) is above a threshold,maxvalue(n)>threshold, then the candidate resize factor that providedthe maximal match score is selected as the resize factor. Else, for asubsequent iteration, the range of possible resize factors is scaled tobe x=[max(0,maxvalue(n)−2*step_(size)),[max(maxvalue(n)+2*step_(size),1)], and the step size is scaled to bestep_size=step_size/10. If the maximal match score in a currentiteration is lower than the maximal match score in a previous iterationmaxvalue(n−1)>maxvalue(n) the algorithm terminates with an error result.If, the maximal match score in a current iteration equals the maximalmatch score in a previous iteration maxvalue(n−1)=maxvalue(n) thealgorithm terminates and the candidate resize factor that provided themaximal match score in the current iteration is selected as the resizefactor.

In operation 2330, the selected resize factor may be used to match thesizes of the anchors to the size of the run-time application image,e.g., to the size of the same anchors in the run-time application image.Depending on the resize factor, matching the sizes of the anchors to thesize of the same anchors in the run-time application image may includeresizing, e.g., zooming out the run-time application image or theanchors using the selected resize factor or the stored anchors, e.g.,the anchors identified in operation 1384 during the design time.

According to embodiments of the invention, the processor may detect if acontrol is editable or not during run-time mode based on a mouse pointedtype. In some embodiments, the processor may detect if a control iseditable or not during run-time mode by hovering or moving an emulatedmouse (e.g., a mouse emulated by the processor as opposed to controlledby a user) over the control, and obtaining cursor information and themouse pointer type. For example, the mouse pointer type may be providedby the operating system. In some embodiments, the mouse pointer typewhen the mouse is located over the control may indicate whether thecontrol is editable or not. For example, in some implementations, if themouse pointer type is “beam”, then the control is editable. Otherwise,the control is not editable.

According to embodiments of the invention, the processor may detect if acontrol is enabled or disabled in run-time mode. As a preparation step,during design time mode, the processor may calculate or find thedominant, representative or average color of the image of the designtime control image, for example, using the GetDominantColor function onthe image of the design time control. During run-time mode, theprocessor may obtain the image of the control, e.g., using the LeptonicaGetWordsInTextlines function. The processor may calculate or find thedominant, representative or average color of the image of the control atrun-time, for example, using the GetDominantColor function. Theprocessor may compare the dominant color of the image of the design timecontrol image with the dominant color of the image of the control atrun-time. If the colors match (e.g., if the difference is lower than athreshold) than it may be concluded that the control is enabled. If,however, the colors do not match (e.g., if the difference is not lowerthan a threshold), than it may be concluded that the control is notenabled.

Unless explicitly stated, the method embodiments described herein arenot constrained to a particular order or sequence. Furthermore, allformulas described herein are intended as examples only and other ordifferent formulas may be used. Some of the described method embodimentsor elements thereof may occur or be performed at the same point in time.

While certain features of the invention have been illustrated anddescribed herein, many modifications, substitutions, changes, andequivalents may occur to those skilled in the art. It is, therefore, tobe understood that the appended claims are intended to cover all suchmodifications and changes as fall within the true spirit of theinvention.

Various embodiments have been presented. Each of these embodiments mayof course include features from other embodiments presented, andembodiments not specifically described may include various featuresdescribed herein.

1. A method for enabling graphic-based interoperability with anapplication executed on a remote computer remote to a local computercomprising a local processor, and displayed on a local display, themethod comprising, using the local processor: during a design-time mode:receiving a design-time application image and a control object locatedwithin the design-time application image; identifying an anchor for thecontrol object; and storing the design-time control object and theassociated anchor; and during a run-time mode: capturing a run-timeapplication image displaying on the local display; identifying theanchor within the run-time application image; and identifying thecontrol object within the run-time application image based on theanchor.
 2. The method of claim 1, wherein the anchor represents a visualobject in the design-time application image having content which remainsconstant from the design-time mode to the run-time mode, and wherein thecontrol object represents a controllable object located within thedesign-time and run-time application images.
 3. The method of claim 1,wherein the control object is one of a button, a link, a check box, aradio button, a scrollbar or a table, wherein: if the control object isa button or a link, then identifying the anchor for the control objectcomprises selecting content image located inside a bounding shape objectthat defines the control. if the control object is a check box or aradio button, then identifying the anchor for the control objectcomprises selecting content image located nearest to the control. if thecontrol object is a scrollbar, then identifying the anchor for thecontrol object comprises selecting at least one image of a scrollbarbutton. if the control object is a table, then identifying the anchorfor the control object comprises selecting image of content of a headercell.
 4. The method of claim 1, wherein identifying the anchor for thecontrol object comprises selecting an image of text or icon that islocated inside a bounding shape object that defines the control or isnearest to the control.
 5. The method of claim 1, comprising: during thedesign-time mode, storing an estimated location of the anchor in thedesign-time application image, wherein identifying the anchor within therun-time application image comprises searching for the anchor startingfrom the estimated location.
 6. The method of claim 5, wherein storingestimated location of the anchor in the design-time application imagecomprises: dividing the design-time application image into segments;identifying the one or more segments containing the anchor; and storingthe identified segments, wherein searching for the anchor starting fromthe estimated location in the run-time application image comprisessearching for the anchor staring in the location of the identifiedsegments.
 7. The method of claim 1, wherein identifying the anchorwithin the run-time application image comprises identifying the visualobject of the anchor within the run-time application image.
 8. Themethod of claim 1, wherein identifying the anchor within the run-timeapplication image comprises: during the run-time mode: calculating aresize factor for the run-time application image or the anchor; matchingthe size of the run-time application image to the size of the visualobject of the anchor using the resize factor; and identifying the visualobject of the anchor within the run-time application image.
 9. Themethod of claim 8, wherein calculating a resize factor comprises:resizing one of the run-time application image or the anchor using aninitial resize factor; matching the resized run-time application imageto the anchor or the resized anchor to the run-time application imageand calculating a match score; repeating resizing and matching withdifferent resize factors until the match score meets a criterion; andselecting the resize factor that provided the match score that meets thecriterion.
 10. The method of claim 8, wherein calculating a resizefactor from a range of possible resize factors comprises: in eachiteration: selecting a step size for generating a plurality of resizefactors within the range of possible resize factors; resizing therun-time application image using the generated resize factors; for eachresize factor, matching the resized run-time application image to theanchor and calculating a match score; reducing the step size; reducingthe range of possible resize factors to be closer, than the range beforereduction, to the resize factor that provided a maximal match score inthe current iteration; repeating the iterations until a maximal matchscore of a current iteration equals a maximal match score of a previousiteration or until the match score is above a threshold; and selectingthe resize factor that provided the maximal match score.
 11. The methodof claim 8, wherein calculating a resize factor comprises: in eachiteration of a first set of iterations: selecting a first step size forgenerating a plurality of first resize factors within a first range ofpossible resize factors; resizing the run-time application image usingthe generated first resize factors; for each first resize factor,matching the resized run-time application image to the anchor andcalculating a match score; if the maximal match score in the currentiteration is lower than the maximal match score in the previousiteration then performing a second set of iterations; else: reducing thefirst step size; reducing the range of possible resize factors to becloser, than the range before resizing, to the resize factor thatprovided a maximal match score in a current iteration; and repeating theiterations until the maximal match score in the current iteration equalsa maximal match score of a previous iteration or until the match scoreis above a threshold; wherein each iteration of the second set ofiterations comprises: selecting a second step size for generating aplurality of second resize factors within a second range of possibleresize factors; resizing the anchor using the generated second resizefactors; for each second resize factor, matching the resized anchor tothe run-time application image and calculating the match score; if themaximal match score in the current iteration is lower than the maximalmatch score in the previous iteration then terminating with an errorresult; else: reducing the second step size; reducing the second rangeof possible resize factors to be closer, than the range before thereduction, to the second resize factor that provided the maximal matchscore in the current iteration; and repeating the iterations of thesecond set of iterations until the maximal match score in the currentiteration equals the maximal match score of a previous iteration oruntil the maximal match score is above the threshold; and selecting oneof the first resize factors or the second resize factors that providedthe maximal match score.
 12. The method of claim 8, wherein calculatinga resize factor comprises: resizing the run-time application image usingan initial resize factor; matching the resized run-time applicationimage to the anchor and calculating a match score; repeating resizingand matching with different resize factors; if a match score that meetsa criterion is found, then selecting the resize factor that provided thematch score that meets the criterion; else: resizing the anchor using aninitial resize factor; matching the resized anchor to the run-timeapplication image and calculating a match score; repeating resizing andmatching with different resize factors; if a match score that meets acriterion is found, then selecting the resize factor that provided thematch score that meets the criterion; and else, terminating with anerror result.
 13. A system for enabling graphic-based interoperabilitywith an application executed on a remote computer, comprising: a localcomputer remote to the remote computer, the local computer comprising: amemory; a local display; and a local processor configured to: during adesign-time mode: display a design-time application image on the localdisplay; capture the design-time application image; receive a controlobject located within the design-time application image; identify ananchor for the control object; and store the design-time control objectand the associated anchor; and during a run-time mode: display arun-time application image on the local display; capture the run-timeapplication image displaying on the local display; identify the anchorwithin the run-time application image; and identify the control objectwithin the run-time application image based on the anchor.
 14. Thesystem of claim 13, wherein the control object is one of a button, alink, a check box, a radio button, a scrollbar or a table, wherein: ifthe control object is a button or a link, then the local processor isconfigured to identify the anchor for the control object by selectingcontent image located inside a bounding shape object that defines thecontrol; if the control object is a check box or a radio button, thenthe local processor is configured to identify the anchor for the controlobject by selecting content image located nearest to the control; if thecontrol object is a scrollbar, then the local processor is configured toidentify the anchor for the control object by selecting at least oneimage of a scrollbar button; and if the control object is a table, thenthe local processor is configured to identify the anchor for the controlobject by selecting image of content of a header cell.
 15. The system ofclaim 13, wherein the local processor is configured to identify theanchor for the control object by selecting an image of text or icon thatis located inside a bounding shape object that defines the control or isnearest to the control.
 16. The system of claim 13, wherein the localprocessor is configured to: during the design-time mode, store anestimated location of the anchor in the design-time application image;and identify the anchor within the run-time application image bysearching for the anchor starting from the estimated location.
 17. Thesystem of claim 16, wherein the local processor is configured to: storethe estimated location of the anchor in the design-time applicationimage by: dividing the design-time application image into segments;identifying the one or more segments containing the anchor; and storingthe identified segments; and search for the anchor starting from theestimated location in the run-time application image by searching forthe anchor staring in the location of the identified segments.
 18. Thesystem of claim 13, wherein the local processor is configured toidentify the anchor within the run-time application image by: during therun-time mode: calculating a resize factor for the run-time applicationimage or the anchor; matching the size of the run-time application imageto the size of the visual object of the anchor using the resize factor;and identifying the visual object of the anchor within the run-timeapplication image.
 19. The system of claim 18, wherein the localprocessor is configured to calculate a resize factor from a range ofpossible resize factors by: in each iteration: selecting a step size forgenerating a plurality of resize factors within the range of possibleresize factors; resizing the run-time application image using thegenerated resize factors; for each resize factor, matching the resizedrun-time application image to the anchor and calculating a match score;reducing the step size; reducing the range of possible resize factors tobe closer to the resize factor that provided a maximal match score inthe current iteration; repeating the iterations until a maximal matchscore of a current iteration equals a maximal match score of a previousiteration or until the match score is above a threshold; and selectingthe resize factor that provided the maximal match score.
 20. The systemof claim 18, wherein the local processor is configured to calculate aresize factor from a range of possible resize factors by: in eachiteration of a first set of iterations: selecting a step size forgenerating a plurality of resize factors within the range of possibleresize factors; resizing the run-time application image using thegenerated resize factors; for each resize factor, matching the resizedrun-time application image to the anchor and calculating a match score;reducing the step size; reducing the range of possible resize factors tobe closer than the range before reducing to the resize factor thatprovided a maximal match score in a current iteration; repeating theiterations until the maximal match score in the current iteration equalsa maximal match score of a previous iteration or until the match scoreis above a threshold or until the maximal match score in the currentiteration is lower than the maximal match score in the previousiteration; if the maximal match score in the current iteration equalsthe maximal match score of the previous iteration or the match score isabove the threshold then selecting the resize factor that provided themaximal match score; and if the maximal match score in the currentiteration is lower than the maximal match score in the previousiteration then performing a second set of iterations, wherein eachiteration of the second set of iterations comprises: selecting a secondstep size for generating a plurality of second resize factors within asecond range of possible resize factors; resizing the anchor using thegenerated second resize factors; for each second resize factor, matchingthe resized anchor to the run-time application image and calculating thematch score; reducing the second step size; reducing the second range ofpossible resize factors to be closer than the range before reducing tothe second resize factor that provided the maximal match score in thecurrent iteration; repeating the iterations of the second set ofiterations until the maximal match score in the current iteration equalsthe maximal match score of a previous iteration or until the maximalmatch score is above the threshold or until the maximal match score inthe current iteration is lower than the maximal match score in theprevious iteration; if the maximal match score in the current iterationequals the maximal match score of the previous iteration or the matchscore is above the threshold then selecting the second resize factorthat provided the maximal match score; and if the maximal match score inthe current iteration is lower than the maximal match score in theprevious iteration then terminating with an error result.